Jul 17 17:24:38.081554 ip-10-0-132-183 systemd[1]: Starting Kubernetes Kubelet... Jul 17 17:24:38.465436 ip-10-0-132-183 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 17:24:38.465436 ip-10-0-132-183 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 17:24:38.465436 ip-10-0-132-183 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 17:24:38.465436 ip-10-0-132-183 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 17:24:38.466122 ip-10-0-132-183 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 17:24:38.466671 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.466497 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 17:24:38.474174 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474149 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:24:38.474174 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474169 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:24:38.474174 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474173 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:24:38.474174 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474176 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:24:38.474174 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474181 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474186 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474190 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474195 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474199 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474202 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474205 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474209 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474212 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474215 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474218 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474221 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474224 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474227 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474229 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474231 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474235 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474237 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474240 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:24:38.474400 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474243 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474245 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474248 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474251 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474253 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474256 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474271 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474274 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474285 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474288 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474290 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474293 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474296 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474332 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474336 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474341 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474345 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474348 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474351 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474354 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:24:38.474851 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474356 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474359 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474361 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474364 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474366 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474369 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474371 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474374 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474376 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474379 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474382 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474384 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474387 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474390 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474392 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474395 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474397 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474400 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474404 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474406 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:24:38.475388 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474409 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474413 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474416 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474418 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474421 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474423 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474426 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474430 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474433 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474436 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474438 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474441 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474444 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474447 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474449 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474452 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474455 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474457 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474460 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474462 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:24:38.475870 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474466 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474469 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474472 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474474 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474477 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474479 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474482 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474484 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474486 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474489 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474491 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474494 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474497 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474499 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474501 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.474504 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475226 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475233 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475236 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:24:38.476363 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475240 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475243 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475247 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475249 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475252 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475255 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475258 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475277 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475280 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475283 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475286 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475289 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475291 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475294 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475297 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475300 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475302 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475305 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475308 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475310 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:24:38.476815 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475313 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475315 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475318 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475321 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475323 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475325 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475328 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475330 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475333 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475335 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475338 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475341 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475344 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475347 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475351 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475353 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475356 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475359 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475362 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475364 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:24:38.477316 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475367 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475369 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475371 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475377 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475381 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475384 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475387 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475390 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475393 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475397 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475401 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475404 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475406 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475409 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475411 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475414 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475417 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475419 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475422 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:24:38.477799 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475424 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475428 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475431 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475433 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475436 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475439 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475443 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475445 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475448 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475450 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475453 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475455 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475458 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475461 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475464 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475466 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475469 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475471 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475474 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:24:38.478247 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475476 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475479 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475482 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475484 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475487 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475491 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475494 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475496 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475499 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475502 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475505 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475508 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475510 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475513 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475515 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475518 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475520 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475525 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475941 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:24:38.478721 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475946 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475949 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475952 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475955 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475958 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475960 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475963 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475966 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475968 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475971 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475973 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475976 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475978 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475981 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475983 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475986 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475989 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475991 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475994 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475996 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:24:38.479208 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.475999 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476001 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476004 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476007 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476009 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476012 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476014 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476016 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476021 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476024 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476029 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476031 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476034 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476037 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476039 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476042 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476044 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476047 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476049 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476054 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:24:38.479705 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476057 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476060 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476063 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476065 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476067 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476070 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476073 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476075 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476078 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476081 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476084 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476087 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476090 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476092 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476094 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476097 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476101 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476104 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476107 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476110 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:24:38.480198 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476112 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476115 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476119 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476122 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476125 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476129 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476132 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476135 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476137 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476140 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476143 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476148 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476151 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476153 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476157 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476160 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476163 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476165 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476170 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:24:38.480727 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476172 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476175 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476177 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476180 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476183 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476185 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476188 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476190 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476193 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476195 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476198 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476201 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476203 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476205 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476208 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476223 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476227 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476230 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476233 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476321 2573 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476336 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 17:24:38.481188 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476342 2573 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476347 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476351 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476355 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476360 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476364 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476367 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476370 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476374 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476377 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476380 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476383 2573 flags.go:64] FLAG: --cgroup-root="" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476386 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476389 2573 flags.go:64] FLAG: --client-ca-file="" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476393 2573 flags.go:64] FLAG: --cloud-provider="external" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476396 2573 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476401 2573 flags.go:64] FLAG: --cluster-domain="" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476404 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476407 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476410 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476413 2573 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476417 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476420 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 17:24:38.481705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476423 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476426 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476429 2573 flags.go:64] FLAG: --contention-profiling="false" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476433 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476436 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476439 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476442 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476446 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476449 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476451 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476454 2573 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476457 2573 flags.go:64] FLAG: --enable-server="true" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476461 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476466 2573 flags.go:64] FLAG: --event-burst="100" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476469 2573 flags.go:64] FLAG: --event-qps="50" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476472 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476475 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476478 2573 flags.go:64] FLAG: --eviction-hard="" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476482 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476484 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476487 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476490 2573 flags.go:64] FLAG: --eviction-soft="" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476493 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476496 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476499 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 17:24:38.482277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476502 2573 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476505 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476508 2573 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476511 2573 flags.go:64] FLAG: --feature-gates="" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476514 2573 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476517 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476520 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476523 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476526 2573 flags.go:64] FLAG: --healthz-port="10248" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476529 2573 flags.go:64] FLAG: --help="false" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476532 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-132-183.ec2.internal" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476535 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476538 2573 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476541 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476544 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476548 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476551 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476553 2573 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476556 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476560 2573 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476563 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476566 2573 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476569 2573 flags.go:64] FLAG: --kube-reserved="" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476572 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 17:24:38.483007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476575 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476577 2573 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476580 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476583 2573 flags.go:64] FLAG: --lock-file="" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476586 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476588 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476591 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476597 2573 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476600 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476603 2573 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476606 2573 flags.go:64] FLAG: --logging-format="text" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476609 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476612 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476615 2573 flags.go:64] FLAG: --manifest-url="" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476618 2573 flags.go:64] FLAG: --manifest-url-header="" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476622 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476625 2573 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476629 2573 flags.go:64] FLAG: --max-pods="110" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476632 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476635 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476637 2573 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476640 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476644 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476646 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476649 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 17:24:38.483604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476656 2573 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476659 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476663 2573 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476666 2573 flags.go:64] FLAG: --pod-cidr="" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476669 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476675 2573 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476678 2573 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476681 2573 flags.go:64] FLAG: --pods-per-core="0" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476684 2573 flags.go:64] FLAG: --port="10250" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476687 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476690 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a8d4a52d893832e7" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476693 2573 flags.go:64] FLAG: --qos-reserved="" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476696 2573 flags.go:64] FLAG: --read-only-port="10255" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476699 2573 flags.go:64] FLAG: --register-node="true" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476702 2573 flags.go:64] FLAG: --register-with-taints="" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476706 2573 flags.go:64] FLAG: --registry-burst="10" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476709 2573 flags.go:64] FLAG: --registry-qps="5" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476712 2573 flags.go:64] FLAG: --reserved-cpus="" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476715 2573 flags.go:64] FLAG: --reserved-memory="" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476718 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476721 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476724 2573 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476726 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476729 2573 flags.go:64] FLAG: --runonce="false" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476732 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 17:24:38.484199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476735 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476738 2573 flags.go:64] FLAG: --seccomp-default="false" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476740 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476743 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476746 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476749 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476752 2573 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476755 2573 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476758 2573 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476762 2573 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476765 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476768 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476771 2573 flags.go:64] FLAG: --system-cgroups="" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476773 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476779 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476781 2573 flags.go:64] FLAG: --tls-cert-file="" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476784 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476788 2573 flags.go:64] FLAG: --tls-min-version="" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476791 2573 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476794 2573 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476797 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476800 2573 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476803 2573 flags.go:64] FLAG: --v="2" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476807 2573 flags.go:64] FLAG: --version="false" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476811 2573 flags.go:64] FLAG: --vmodule="" Jul 17 17:24:38.484806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476815 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.476818 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476937 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476941 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476944 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476946 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476949 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476951 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476956 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476959 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476961 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476964 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476966 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476970 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476974 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476977 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476981 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476984 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476987 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476990 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:24:38.485484 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476993 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476996 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.476998 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477001 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477004 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477006 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477009 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477011 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477014 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477016 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477019 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477021 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477024 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477027 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477029 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477032 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477034 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477037 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477039 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477042 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:24:38.485976 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477046 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477049 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477051 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477054 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477056 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477059 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477061 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477064 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477069 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477071 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477074 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477078 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477081 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477084 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477087 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477090 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477093 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477096 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477098 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:24:38.486483 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477101 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477104 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477107 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477110 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477112 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477114 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477117 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477120 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477122 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477125 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477128 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477130 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477133 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477137 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477140 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477142 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477145 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477147 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477150 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:24:38.486953 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477153 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477155 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477161 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477164 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477167 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477169 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477172 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477174 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477177 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477180 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477184 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477187 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477190 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477192 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477195 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477197 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477200 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477202 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477205 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477207 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:24:38.487433 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477211 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:24:38.487916 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477213 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:24:38.487916 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.477216 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:24:38.487916 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.477838 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 17:24:38.487916 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.484586 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 17:24:38.487916 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.484601 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 17:24:38.487916 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.484620 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 17:24:38.487916 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.484625 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 17:24:38.487916 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484680 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:24:38.487916 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484686 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:24:38.487916 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484689 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:24:38.487916 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484692 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:24:38.487916 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484695 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:24:38.487916 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484698 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:24:38.487916 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484701 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:24:38.487916 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484704 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484706 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484709 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484712 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484714 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484717 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484721 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484725 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484728 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484731 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484734 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484737 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484740 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484742 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484745 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484748 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484751 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484753 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484756 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:24:38.488321 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484758 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484762 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484766 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484774 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484777 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484780 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484782 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484785 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484788 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484790 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484793 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484795 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484798 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484801 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484803 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484806 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484809 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484811 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484814 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:24:38.488778 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484817 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484819 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484822 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484825 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484827 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484830 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484832 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484835 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484837 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484840 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484842 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484845 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484847 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484850 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484852 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484855 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484859 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484862 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484865 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484868 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:24:38.489244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484871 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484874 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484878 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484881 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484883 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484886 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484889 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484892 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484895 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484897 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484900 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484902 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484905 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484907 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484910 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484912 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484915 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484917 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484920 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:24:38.489866 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484922 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:24:38.490350 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484925 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:24:38.490350 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484928 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:24:38.490350 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484930 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:24:38.490350 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484933 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:24:38.490350 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484935 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:24:38.490350 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484938 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:24:38.490350 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484940 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:24:38.490350 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484943 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:24:38.490350 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484946 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:24:38.490350 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484949 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:24:38.490350 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484952 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:24:38.490350 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484955 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:24:38.490350 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484957 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:24:38.490350 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.484960 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:24:38.490350 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.484964 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485078 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485083 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485086 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485089 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485091 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485094 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485096 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485099 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485102 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485104 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485107 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485109 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485112 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485114 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485117 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485119 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485122 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485124 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485127 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:24:38.490722 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485130 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485133 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485135 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485138 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485140 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485143 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485145 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485148 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485151 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485158 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485161 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485163 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485166 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485169 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485171 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485174 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485176 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485179 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485181 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485184 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:24:38.491210 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485186 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485188 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485191 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485193 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485196 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485198 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485201 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485203 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485206 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485209 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485211 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485214 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485216 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485221 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485234 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485238 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485241 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485244 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485248 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485250 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:24:38.491697 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485253 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485256 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485273 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485278 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485281 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485284 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485287 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485289 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485292 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485294 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485297 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485299 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485302 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485304 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485307 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485310 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485312 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485314 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485317 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485320 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:24:38.492187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485322 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485325 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485328 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485330 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485333 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485335 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485338 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485340 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485343 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485345 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485350 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485353 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485356 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485359 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485362 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485366 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485369 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485371 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485374 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:24:38.492682 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:38.485376 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:24:38.493133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.485382 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 17:24:38.493133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.485500 2573 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 17:24:38.493133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.487421 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 17:24:38.493133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.488651 2573 server.go:1019] "Starting client certificate rotation" Jul 17 17:24:38.493133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.488750 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 17:24:38.493133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.489772 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 17:24:38.493133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.491051 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 17:24:38.497572 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.497550 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 17:24:38.511636 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.511618 2573 log.go:25] "Validated CRI v1 runtime API" Jul 17 17:24:38.513713 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.513699 2573 log.go:25] "Validated CRI v1 image API" Jul 17 17:24:38.515650 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.515634 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 17:24:38.519322 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.519291 2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 af9317c0-4328-4e41-b427-3e9498ca9bd7:/dev/nvme0n1p3 be0437d9-f232-4fa5-9195-f9939259abb4:/dev/nvme0n1p4] Jul 17 17:24:38.519416 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.519321 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 17:24:38.523433 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.523414 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 17:24:38.524608 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.524497 2573 manager.go:216] Machine: {Timestamp:2026-07-17 17:24:38.523013609 +0000 UTC m=+0.329794747 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3074855 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20aa260a8a71269326136deed9d2f0 SystemUUID:ec20aa26-0a8a-7126-9326-136deed9d2f0 BootID:06a25062-34cc-4e15-9d57-d598a10a58e9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:10:98:5e:36:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:10:98:5e:36:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:4c:c6:29:3f:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 17:24:38.524608 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.524604 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 17:24:38.524770 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.524709 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 17:24:38.527708 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.527679 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 17:24:38.527850 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.527710 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-183.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 17:24:38.529163 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.529153 2573 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 17:24:38.529197 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.529164 2573 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 17:24:38.529197 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.529172 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 17:24:38.530133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.530122 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 17:24:38.531648 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.531639 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 17:24:38.534913 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.534901 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 17 17:24:38.535027 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.535017 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 17:24:38.535068 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.535047 2573 kubelet.go:486] "Attempting to sync node with API server" Jul 17 17:24:38.535068 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.535056 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 17:24:38.535068 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.535066 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 17:24:38.535169 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.535100 2573 kubelet.go:398] "Adding apiserver pod source" Jul 17 17:24:38.535169 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.535108 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 17:24:38.535524 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.535512 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 17:24:38.535567 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.535531 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 17:24:38.537994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.537974 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 17:24:38.539622 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.539610 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 17:24:38.539665 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.539640 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 17:24:38.540326 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.540316 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 17:24:38.540369 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.540332 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 17:24:38.540369 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.540338 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 17:24:38.540369 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.540343 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 17:24:38.540369 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.540351 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 17:24:38.540369 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.540358 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 17:24:38.540671 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.540650 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 17:24:38.540671 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.540665 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 17:24:38.540735 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.540675 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 17:24:38.540735 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.540684 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 17:24:38.540735 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.540696 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 17:24:38.540735 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.540711 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 17:24:38.542431 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.542419 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 17:24:38.542466 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.542432 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 17:24:38.545923 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.545892 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j98ww" Jul 17 17:24:38.546150 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.546136 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-183.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 17:24:38.546193 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.546170 2573 server.go:1268] "Started kubelet" Jul 17 17:24:38.546592 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.546541 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 17:24:38.546592 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.546557 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 17:24:38.546720 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.546592 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 17:24:38.547090 ip-10-0-132-183 systemd[1]: Started Kubernetes Kubelet. Jul 17 17:24:38.548765 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.548725 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 17:24:38.548845 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.548799 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 17:24:38.550491 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.550471 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 17:24:38.550934 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.550906 2573 server.go:310] "Adding debug handlers to kubelet server" Jul 17 17:24:38.551688 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.551662 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 17:24:38.553397 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.552457 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-183.ec2.internal.18c323a73c3f5125 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-183.ec2.internal,UID:ip-10-0-132-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-183.ec2.internal,},FirstTimestamp:2026-07-17 17:24:38.546141477 +0000 UTC m=+0.352922616,LastTimestamp:2026-07-17 17:24:38.546141477 +0000 UTC m=+0.352922616,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-183.ec2.internal,}" Jul 17 17:24:38.555449 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.555436 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j98ww" Jul 17 17:24:38.556356 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.556338 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 17:24:38.556449 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.556341 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 17:24:38.556699 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.556602 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 17 17:24:38.557170 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.557154 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 17:24:38.557255 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.557189 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 17:24:38.557343 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.557236 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 17:24:38.557343 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.557214 2573 factory.go:55] Registering systemd factory Jul 17 17:24:38.557431 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.557345 2573 factory.go:223] Registration of the systemd container factory successfully Jul 17 17:24:38.557431 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.557398 2573 reconstruct.go:97] "Volume reconstruction finished" Jul 17 17:24:38.557431 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.557407 2573 reconciler.go:29] "Reconciler: start to sync state" Jul 17 17:24:38.557564 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.557551 2573 factory.go:153] Registering CRI-O factory Jul 17 17:24:38.557599 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.557568 2573 factory.go:223] Registration of the crio container factory successfully Jul 17 17:24:38.557658 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.557615 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 17:24:38.557658 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.557636 2573 factory.go:103] Registering Raw factory Jul 17 17:24:38.557658 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.557650 2573 manager.go:1196] Started watching for new ooms in manager Jul 17 17:24:38.558390 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.558375 2573 manager.go:349] Starting recovery of all containers Jul 17 17:24:38.566947 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.566892 2573 manager.go:354] Recovery completed Jul 17 17:24:38.567055 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.567027 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 17 17:24:38.567160 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.567132 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 17 17:24:38.569079 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.568961 2573 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 17 17:24:38.572095 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.572083 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:24:38.574147 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.574133 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:24:38.574223 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.574160 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:24:38.574223 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.574173 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:24:38.574667 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.574655 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 17:24:38.574667 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.574666 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 17:24:38.574749 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.574684 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 17 17:24:38.574749 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.574724 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:24:38.575457 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.575443 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:24:38.575494 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.575468 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:24:38.575494 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.575477 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:24:38.575841 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.575831 2573 policy_none.go:49] "None policy: Start" Jul 17 17:24:38.575872 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.575846 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 17:24:38.575872 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.575856 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 17:24:38.576771 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.576762 2573 policy_none.go:47] "Start" Jul 17 17:24:38.609829 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.609810 2573 manager.go:337] "Starting Device Plugin manager" Jul 17 17:24:38.609943 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.609865 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 17:24:38.609943 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.609874 2573 server.go:87] "Starting device plugin registration server" Jul 17 17:24:38.610112 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.610100 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 17:24:38.610146 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.610114 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 17:24:38.610220 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.610202 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 17:24:38.610313 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.610302 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 17:24:38.610313 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.610313 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 17:24:38.610759 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.610739 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 17:24:38.610837 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.610784 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-183.ec2.internal\" not found" Jul 17 17:24:38.698277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.698235 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 17:24:38.699367 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.699352 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 17:24:38.699446 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.699396 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 17:24:38.699446 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.699425 2573 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 17:24:38.699534 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.699470 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 17:24:38.701967 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.701947 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:24:38.710880 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.710860 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:24:38.711684 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.711668 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:24:38.711753 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.711704 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:24:38.711753 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.711721 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:24:38.711753 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.711746 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-183.ec2.internal" Jul 17 17:24:38.717473 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.717427 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-183.ec2.internal" Jul 17 17:24:38.717473 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.717450 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-183.ec2.internal\": node \"ip-10-0-132-183.ec2.internal\" not found" Jul 17 17:24:38.747947 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.747923 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 17 17:24:38.800400 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.800373 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal"] Jul 17 17:24:38.800508 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.800452 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:24:38.802120 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.802103 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:24:38.802207 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.802129 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:24:38.802207 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.802138 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:24:38.803531 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.803519 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:24:38.803683 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.803670 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 17 17:24:38.803725 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.803717 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:24:38.804236 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.804224 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:24:38.804236 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.804230 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:24:38.804368 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.804243 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:24:38.804368 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.804251 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:24:38.804368 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.804253 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:24:38.804368 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.804276 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:24:38.805546 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.805530 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" Jul 17 17:24:38.805657 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.805559 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:24:38.806172 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.806155 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:24:38.806254 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.806178 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:24:38.806254 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.806187 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:24:38.831892 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.831875 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-183.ec2.internal\" not found" node="ip-10-0-132-183.ec2.internal" Jul 17 17:24:38.836290 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.836276 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-183.ec2.internal\" not found" node="ip-10-0-132-183.ec2.internal" Jul 17 17:24:38.848610 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.848591 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 17 17:24:38.858910 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.858892 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c4336710fed4961d5d53b67aadb1cb02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal\" (UID: \"c4336710fed4961d5d53b67aadb1cb02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 17 17:24:38.858959 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.858916 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4336710fed4961d5d53b67aadb1cb02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal\" (UID: \"c4336710fed4961d5d53b67aadb1cb02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 17 17:24:38.858959 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.858933 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e2d2ffede5ac360572d29fe6cd09b1b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-183.ec2.internal\" (UID: \"3e2d2ffede5ac360572d29fe6cd09b1b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" Jul 17 17:24:38.948959 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:38.948928 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 17 17:24:38.959308 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.959286 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e2d2ffede5ac360572d29fe6cd09b1b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-183.ec2.internal\" (UID: \"3e2d2ffede5ac360572d29fe6cd09b1b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" Jul 17 17:24:38.959359 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.959314 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c4336710fed4961d5d53b67aadb1cb02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal\" (UID: \"c4336710fed4961d5d53b67aadb1cb02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 17 17:24:38.959359 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.959331 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4336710fed4961d5d53b67aadb1cb02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal\" (UID: \"c4336710fed4961d5d53b67aadb1cb02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 17 17:24:38.959416 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.959387 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4336710fed4961d5d53b67aadb1cb02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal\" (UID: \"c4336710fed4961d5d53b67aadb1cb02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 17 17:24:38.959493 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.959471 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e2d2ffede5ac360572d29fe6cd09b1b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-183.ec2.internal\" (UID: \"3e2d2ffede5ac360572d29fe6cd09b1b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" Jul 17 17:24:38.959550 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:38.959521 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c4336710fed4961d5d53b67aadb1cb02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal\" (UID: \"c4336710fed4961d5d53b67aadb1cb02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 17 17:24:39.049740 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:39.049673 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 17 17:24:39.134244 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.134213 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 17 17:24:39.140141 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.140119 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" Jul 17 17:24:39.149954 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:39.149932 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 17 17:24:39.250569 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:39.250518 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 17 17:24:39.351089 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:39.351004 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 17 17:24:39.451595 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:39.451554 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-183.ec2.internal\" not found" Jul 17 17:24:39.488071 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.488047 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 17:24:39.488460 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.488170 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 17:24:39.524888 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.524865 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:24:39.535857 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.535833 2573 apiserver.go:52] "Watching apiserver" Jul 17 17:24:39.545879 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.545859 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 17:24:39.546806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.546785 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-wjj97","openshift-multus/multus-dqkbz","openshift-multus/network-metrics-daemon-w49zh","openshift-network-diagnostics/network-check-target-pvv46","openshift-network-operator/iptables-alerter-s89r6","openshift-ovn-kubernetes/ovnkube-node-7lgg2","kube-system/global-pull-secret-syncer-hr9d5","kube-system/konnectivity-agent-bk7mr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx","openshift-cluster-node-tuning-operator/tuned-flc96","openshift-dns/node-resolver-9bwpz","openshift-image-registry/node-ca-mmd7j"] Jul 17 17:24:39.548258 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.548244 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.550644 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.550616 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.551382 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.551362 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 17:24:39.551474 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.551406 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wxwbd\"" Jul 17 17:24:39.551547 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.551481 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 17:24:39.551650 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.551636 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 17:24:39.551713 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.551698 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 17:24:39.551815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.551793 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:39.551929 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:39.551875 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:24:39.552141 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.552123 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 17:24:39.553086 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.553068 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s89r6" Jul 17 17:24:39.553151 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.553107 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 17:24:39.553256 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.553110 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qvf5v\"" Jul 17 17:24:39.554465 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.554450 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.555693 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.555676 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.556203 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.556189 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7fz46\"" Jul 17 17:24:39.556317 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.556303 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 17:24:39.556466 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.556454 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 17:24:39.556611 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.556592 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" Jul 17 17:24:39.556719 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.556703 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 17:24:39.556848 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.556830 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 17:24:39.556937 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.556913 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mmd7j" Jul 17 17:24:39.557073 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.557055 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 17:24:39.557146 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.557073 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 17:24:39.557351 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.557336 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b664h\"" Jul 17 17:24:39.557433 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.557406 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 17:24:39.557492 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.557442 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 17:24:39.557492 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.557452 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 17:24:39.557492 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.557459 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 17:24:39.557977 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.557958 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4szdp\"" Jul 17 17:24:39.558064 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.557984 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 17:24:39.558064 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.558042 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 17:24:39.558362 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.558343 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:39.558445 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:39.558401 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:24:39.559272 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.559246 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 17:24:39.559540 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.559525 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 17:24:39.559610 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.559592 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 17:24:39.559610 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.559604 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:39.559707 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.559660 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5dwm4\"" Jul 17 17:24:39.559753 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:39.559654 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:24:39.560959 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.560944 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bk7mr" Jul 17 17:24:39.562064 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562044 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-run-systemd\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.562146 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562068 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-lib-modules\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.562146 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562085 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-cni-binary-copy\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.562146 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-etc-kubernetes\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.562146 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562116 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e92dd775-d49f-47b9-bf80-448ae55bb6f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.562146 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e92dd775-d49f-47b9-bf80-448ae55bb6f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.562417 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562177 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-sysctl-d\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.562417 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562224 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-host\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.562417 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562256 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-os-release\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.562417 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562285 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.562417 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562298 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-run-netns\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.562417 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562345 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-var-lib-cni-multus\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.562417 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562383 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e92dd775-d49f-47b9-bf80-448ae55bb6f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.562715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-log-socket\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.562715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562452 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-systemd\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.562715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562486 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-multus-socket-dir-parent\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.562715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562509 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-var-lib-kubelet\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.562715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562530 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwnqd\" (UniqueName: \"kubernetes.io/projected/6ca760ea-272f-4b74-b3ff-958d46cffced-kube-api-access-gwnqd\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.562715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562547 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-system-cni-dir\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.562715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562566 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ca760ea-272f-4b74-b3ff-958d46cffced-ovnkube-config\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.562715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562580 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-run-openvswitch\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.562715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562596 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwh42\" (UniqueName: \"kubernetes.io/projected/e97089fa-7036-4cf6-a506-e292a28960d9-kube-api-access-mwh42\") pod \"iptables-alerter-s89r6\" (UID: \"e97089fa-7036-4cf6-a506-e292a28960d9\") " pod="openshift-network-operator/iptables-alerter-s89r6" Jul 17 17:24:39.562715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562614 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-var-lib-openvswitch\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.562715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562642 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-run-multus-certs\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.562715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562660 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e92dd775-d49f-47b9-bf80-448ae55bb6f2-os-release\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.562715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562674 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-run-netns\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.562715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562697 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-etc-openvswitch\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.562715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.563353 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562734 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ca760ea-272f-4b74-b3ff-958d46cffced-ovn-node-metrics-cert\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.563353 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562755 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e92dd775-d49f-47b9-bf80-448ae55bb6f2-system-cni-dir\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.563353 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562792 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ca760ea-272f-4b74-b3ff-958d46cffced-ovnkube-script-lib\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.563353 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562828 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b905e39a-a951-4f8a-9479-348a1fea69e0-host\") pod \"node-ca-mmd7j\" (UID: \"b905e39a-a951-4f8a-9479-348a1fea69e0\") " pod="openshift-image-registry/node-ca-mmd7j" Jul 17 17:24:39.563353 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562857 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b905e39a-a951-4f8a-9479-348a1fea69e0-serviceca\") pod \"node-ca-mmd7j\" (UID: \"b905e39a-a951-4f8a-9479-348a1fea69e0\") " pod="openshift-image-registry/node-ca-mmd7j" Jul 17 17:24:39.563353 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562881 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-multus-cni-dir\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.563353 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562896 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-cnibin\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.563353 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562909 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-node-log\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.563353 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562924 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-run\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.563353 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9vnn\" (UniqueName: \"kubernetes.io/projected/b905e39a-a951-4f8a-9479-348a1fea69e0-kube-api-access-q9vnn\") pod \"node-ca-mmd7j\" (UID: \"b905e39a-a951-4f8a-9479-348a1fea69e0\") " pod="openshift-image-registry/node-ca-mmd7j" Jul 17 17:24:39.563353 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562979 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs\") pod \"network-metrics-daemon-w49zh\" (UID: \"c0ca305d-1ff8-4cab-bde6-bc63609387aa\") " pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:39.563353 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.562994 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzkcb\" (UniqueName: \"kubernetes.io/projected/c0ca305d-1ff8-4cab-bde6-bc63609387aa-kube-api-access-nzkcb\") pod \"network-metrics-daemon-w49zh\" (UID: \"c0ca305d-1ff8-4cab-bde6-bc63609387aa\") " pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:39.563353 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-multus-conf-dir\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.563353 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563053 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e92dd775-d49f-47b9-bf80-448ae55bb6f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.563353 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563083 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-slash\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.563353 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563107 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ca760ea-272f-4b74-b3ff-958d46cffced-env-overrides\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563130 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-modprobe-d\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563161 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-kubernetes\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563186 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-run-k8s-cni-cncf-io\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563208 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-var-lib-cni-bin\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563213 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9d4rp\"" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563225 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563230 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e92dd775-d49f-47b9-bf80-448ae55bb6f2-cnibin\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563132 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563254 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pctx5\" (UniqueName: \"kubernetes.io/projected/e92dd775-d49f-47b9-bf80-448ae55bb6f2-kube-api-access-pctx5\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563377 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-cni-bin\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563403 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e97089fa-7036-4cf6-a506-e292a28960d9-host-slash\") pod \"iptables-alerter-s89r6\" (UID: \"e97089fa-7036-4cf6-a506-e292a28960d9\") " pod="openshift-network-operator/iptables-alerter-s89r6" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563426 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-sysctl-conf\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-multus-daemon-config\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563472 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-sysconfig\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563494 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-var-lib-kubelet\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563515 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsfqg\" (UniqueName: \"kubernetes.io/projected/4d001a36-a8bc-428a-993d-b3418c60b000-kube-api-access-hsfqg\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563569 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9bwpz" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563576 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-run-ovn\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.563996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563618 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-cni-netd\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.564715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563642 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e97089fa-7036-4cf6-a506-e292a28960d9-iptables-alerter-script\") pod \"iptables-alerter-s89r6\" (UID: \"e97089fa-7036-4cf6-a506-e292a28960d9\") " pod="openshift-network-operator/iptables-alerter-s89r6" Jul 17 17:24:39.564715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563665 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-kubelet\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.564715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563688 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-hostroot\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.564715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-run-ovn-kubernetes\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.564715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563733 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d001a36-a8bc-428a-993d-b3418c60b000-tmp\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.564715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563768 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vg9t\" (UniqueName: \"kubernetes.io/projected/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-kube-api-access-2vg9t\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.564715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563792 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v77pt\" (UniqueName: \"kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt\") pod \"network-check-target-pvv46\" (UID: \"b2859ee6-181c-4e9c-a3f6-777df52447fa\") " pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:39.564715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563815 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-systemd-units\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.564715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563841 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-sys\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.564715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.563862 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d001a36-a8bc-428a-993d-b3418c60b000-etc-tuned\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.564715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.564358 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 17:24:39.564715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.564499 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 17:24:39.564715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.564675 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 17:24:39.565241 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.564949 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vdvs5\"" Jul 17 17:24:39.566098 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.566077 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 17:24:39.566203 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.566189 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 17:24:39.566546 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.566527 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8tj6n\"" Jul 17 17:24:39.567806 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.567743 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 17:19:39 +0000 UTC" deadline="2028-04-17 18:05:03.479022746 +0000 UTC" Jul 17 17:24:39.568039 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.568026 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15360h40m23.911001175s" Jul 17 17:24:39.568173 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.568158 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 17:24:39.569709 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.569691 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 17:24:39.569785 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.569757 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" Jul 17 17:24:39.569785 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.569761 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal"] Jul 17 17:24:39.576435 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.576419 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal"] Jul 17 17:24:39.577039 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.577023 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 17:24:39.613832 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.613813 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qbnlj" Jul 17 17:24:39.621846 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.621824 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qbnlj" Jul 17 17:24:39.657763 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.657710 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 17:24:39.664305 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664283 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-etc-selinux\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.664416 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664312 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-socket-dir\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.664416 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664334 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-run-systemd\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.664416 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664375 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-run-systemd\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.664416 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-lib-modules\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.664609 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664431 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-cni-binary-copy\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.664609 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664450 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-etc-kubernetes\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.664609 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e92dd775-d49f-47b9-bf80-448ae55bb6f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.664609 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664475 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-lib-modules\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.664609 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664482 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e92dd775-d49f-47b9-bf80-448ae55bb6f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.664609 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664514 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-etc-kubernetes\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.664609 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664543 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-sysctl-d\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.664609 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664572 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-host\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.664609 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664593 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-os-release\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664615 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-run-netns\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-var-lib-cni-multus\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664669 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-sys-fs\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664657 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-sysctl-d\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664698 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e92dd775-d49f-47b9-bf80-448ae55bb6f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-log-socket\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664737 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-os-release\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664746 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-run-netns\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664742 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-var-lib-cni-multus\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664748 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-systemd\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664795 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-systemd\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664794 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-log-socket\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664797 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-multus-socket-dir-parent\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664830 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-var-lib-kubelet\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664832 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-host\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664855 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.665078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664863 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e92dd775-d49f-47b9-bf80-448ae55bb6f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664883 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvd86\" (UniqueName: \"kubernetes.io/projected/e02547e4-091d-40e4-9ed5-321f22875661-kube-api-access-tvd86\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664913 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17bd4e15-f627-4ce8-8992-9e4e0a754e32-tmp-dir\") pod \"node-resolver-9bwpz\" (UID: \"17bd4e15-f627-4ce8-8992-9e4e0a754e32\") " pod="openshift-dns/node-resolver-9bwpz" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664918 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-multus-socket-dir-parent\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664902 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-var-lib-kubelet\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwnqd\" (UniqueName: \"kubernetes.io/projected/6ca760ea-272f-4b74-b3ff-958d46cffced-kube-api-access-gwnqd\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-system-cni-dir\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.664996 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-system-cni-dir\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665072 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-dbus\") pod \"global-pull-secret-syncer-hr9d5\" (UID: \"2a96d8c8-87eb-47c6-b427-0486a1b6eb23\") " pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665092 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e92dd775-d49f-47b9-bf80-448ae55bb6f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665099 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e92dd775-d49f-47b9-bf80-448ae55bb6f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665099 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-cni-binary-copy\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ca760ea-272f-4b74-b3ff-958d46cffced-ovnkube-config\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665142 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-run-openvswitch\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665230 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-run-openvswitch\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665235 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwh42\" (UniqueName: \"kubernetes.io/projected/e97089fa-7036-4cf6-a506-e292a28960d9-kube-api-access-mwh42\") pod \"iptables-alerter-s89r6\" (UID: \"e97089fa-7036-4cf6-a506-e292a28960d9\") " pod="openshift-network-operator/iptables-alerter-s89r6" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665288 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-var-lib-openvswitch\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.665994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665313 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-run-multus-certs\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665328 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-var-lib-openvswitch\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665343 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p8v8\" (UniqueName: \"kubernetes.io/projected/17bd4e15-f627-4ce8-8992-9e4e0a754e32-kube-api-access-6p8v8\") pod \"node-resolver-9bwpz\" (UID: \"17bd4e15-f627-4ce8-8992-9e4e0a754e32\") " pod="openshift-dns/node-resolver-9bwpz" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665367 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-run-multus-certs\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e92dd775-d49f-47b9-bf80-448ae55bb6f2-os-release\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665396 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-run-netns\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665419 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-etc-openvswitch\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665456 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e92dd775-d49f-47b9-bf80-448ae55bb6f2-os-release\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665457 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665493 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ca760ea-272f-4b74-b3ff-958d46cffced-ovn-node-metrics-cert\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665510 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-run-netns\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665520 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-registration-dir\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665559 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-etc-openvswitch\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e92dd775-d49f-47b9-bf80-448ae55bb6f2-system-cni-dir\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665583 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ca760ea-272f-4b74-b3ff-958d46cffced-ovnkube-config\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665593 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ca760ea-272f-4b74-b3ff-958d46cffced-ovnkube-script-lib\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665582 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.666815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665621 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b905e39a-a951-4f8a-9479-348a1fea69e0-host\") pod \"node-ca-mmd7j\" (UID: \"b905e39a-a951-4f8a-9479-348a1fea69e0\") " pod="openshift-image-registry/node-ca-mmd7j" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665624 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e92dd775-d49f-47b9-bf80-448ae55bb6f2-system-cni-dir\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665657 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b905e39a-a951-4f8a-9479-348a1fea69e0-serviceca\") pod \"node-ca-mmd7j\" (UID: \"b905e39a-a951-4f8a-9479-348a1fea69e0\") " pod="openshift-image-registry/node-ca-mmd7j" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665673 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b905e39a-a951-4f8a-9479-348a1fea69e0-host\") pod \"node-ca-mmd7j\" (UID: \"b905e39a-a951-4f8a-9479-348a1fea69e0\") " pod="openshift-image-registry/node-ca-mmd7j" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665683 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-multus-cni-dir\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665708 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-cnibin\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665732 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/747c8133-308b-4cf0-99cd-d692c4a248ad-agent-certs\") pod \"konnectivity-agent-bk7mr\" (UID: \"747c8133-308b-4cf0-99cd-d692c4a248ad\") " pod="kube-system/konnectivity-agent-bk7mr" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665759 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-node-log\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665780 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-run\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665803 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9vnn\" (UniqueName: \"kubernetes.io/projected/b905e39a-a951-4f8a-9479-348a1fea69e0-kube-api-access-q9vnn\") pod \"node-ca-mmd7j\" (UID: \"b905e39a-a951-4f8a-9479-348a1fea69e0\") " pod="openshift-image-registry/node-ca-mmd7j" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665827 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs\") pod \"network-metrics-daemon-w49zh\" (UID: \"c0ca305d-1ff8-4cab-bde6-bc63609387aa\") " pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665853 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-node-log\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665852 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkcb\" (UniqueName: \"kubernetes.io/projected/c0ca305d-1ff8-4cab-bde6-bc63609387aa-kube-api-access-nzkcb\") pod \"network-metrics-daemon-w49zh\" (UID: \"c0ca305d-1ff8-4cab-bde6-bc63609387aa\") " pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665853 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665893 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-multus-conf-dir\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665918 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-device-dir\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665945 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e92dd775-d49f-47b9-bf80-448ae55bb6f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665971 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-slash\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.667688 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665986 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-run\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665994 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ca760ea-272f-4b74-b3ff-958d46cffced-env-overrides\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-modprobe-d\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666073 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-kubernetes\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666101 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-run-k8s-cni-cncf-io\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666117 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ca760ea-272f-4b74-b3ff-958d46cffced-ovnkube-script-lib\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-var-lib-cni-bin\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666151 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-kubelet-config\") pod \"global-pull-secret-syncer-hr9d5\" (UID: \"2a96d8c8-87eb-47c6-b427-0486a1b6eb23\") " pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666173 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e92dd775-d49f-47b9-bf80-448ae55bb6f2-cnibin\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666238 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pctx5\" (UniqueName: \"kubernetes.io/projected/e92dd775-d49f-47b9-bf80-448ae55bb6f2-kube-api-access-pctx5\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666217 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-modprobe-d\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666309 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-cnibin\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.665945 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-multus-cni-dir\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666361 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-var-lib-cni-bin\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666346 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ca760ea-272f-4b74-b3ff-958d46cffced-env-overrides\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666403 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-kubernetes\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666423 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-multus-conf-dir\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666424 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e92dd775-d49f-47b9-bf80-448ae55bb6f2-cnibin\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.668559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666439 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-host-run-k8s-cni-cncf-io\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666465 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-slash\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:39.666539 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666556 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-cni-bin\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666574 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b905e39a-a951-4f8a-9479-348a1fea69e0-serviceca\") pod \"node-ca-mmd7j\" (UID: \"b905e39a-a951-4f8a-9479-348a1fea69e0\") " pod="openshift-image-registry/node-ca-mmd7j" Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666586 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret\") pod \"global-pull-secret-syncer-hr9d5\" (UID: \"2a96d8c8-87eb-47c6-b427-0486a1b6eb23\") " pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666615 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17bd4e15-f627-4ce8-8992-9e4e0a754e32-hosts-file\") pod \"node-resolver-9bwpz\" (UID: \"17bd4e15-f627-4ce8-8992-9e4e0a754e32\") " pod="openshift-dns/node-resolver-9bwpz" Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666631 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-cni-bin\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:39.666643 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs podName:c0ca305d-1ff8-4cab-bde6-bc63609387aa nodeName:}" failed. No retries permitted until 2026-07-17 17:24:40.166621959 +0000 UTC m=+1.973403086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs") pod "network-metrics-daemon-w49zh" (UID: "c0ca305d-1ff8-4cab-bde6-bc63609387aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666687 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e97089fa-7036-4cf6-a506-e292a28960d9-host-slash\") pod \"iptables-alerter-s89r6\" (UID: \"e97089fa-7036-4cf6-a506-e292a28960d9\") " pod="openshift-network-operator/iptables-alerter-s89r6" Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666713 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-sysctl-conf\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666737 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-multus-daemon-config\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666757 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e97089fa-7036-4cf6-a506-e292a28960d9-host-slash\") pod \"iptables-alerter-s89r6\" (UID: \"e97089fa-7036-4cf6-a506-e292a28960d9\") " pod="openshift-network-operator/iptables-alerter-s89r6" Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-sysconfig\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666809 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-var-lib-kubelet\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666833 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsfqg\" (UniqueName: \"kubernetes.io/projected/4d001a36-a8bc-428a-993d-b3418c60b000-kube-api-access-hsfqg\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666857 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-sysconfig\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.669459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666882 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-var-lib-kubelet\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666912 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-run-ovn\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666933 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-cni-netd\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666954 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e97089fa-7036-4cf6-a506-e292a28960d9-iptables-alerter-script\") pod \"iptables-alerter-s89r6\" (UID: \"e97089fa-7036-4cf6-a506-e292a28960d9\") " pod="openshift-network-operator/iptables-alerter-s89r6" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666974 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-etc-sysctl-conf\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.666977 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-kubelet\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667006 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-kubelet\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667005 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-hostroot\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667035 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-hostroot\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667072 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-run-ovn-kubernetes\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667090 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-cni-netd\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667104 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-run-ovn\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667037 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-host-run-ovn-kubernetes\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667144 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d001a36-a8bc-428a-993d-b3418c60b000-tmp\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667186 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vg9t\" (UniqueName: \"kubernetes.io/projected/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-kube-api-access-2vg9t\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667212 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77pt\" (UniqueName: \"kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt\") pod \"network-check-target-pvv46\" (UID: \"b2859ee6-181c-4e9c-a3f6-777df52447fa\") " pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667240 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/747c8133-308b-4cf0-99cd-d692c4a248ad-konnectivity-ca\") pod \"konnectivity-agent-bk7mr\" (UID: \"747c8133-308b-4cf0-99cd-d692c4a248ad\") " pod="kube-system/konnectivity-agent-bk7mr" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667288 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-systemd-units\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.670179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667311 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-sys\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.670950 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667329 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e92dd775-d49f-47b9-bf80-448ae55bb6f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.670950 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667334 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d001a36-a8bc-428a-993d-b3418c60b000-etc-tuned\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.670950 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667406 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-multus-daemon-config\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.670950 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667646 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e97089fa-7036-4cf6-a506-e292a28960d9-iptables-alerter-script\") pod \"iptables-alerter-s89r6\" (UID: \"e97089fa-7036-4cf6-a506-e292a28960d9\") " pod="openshift-network-operator/iptables-alerter-s89r6" Jul 17 17:24:39.670950 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667678 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ca760ea-272f-4b74-b3ff-958d46cffced-systemd-units\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.670950 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.667692 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d001a36-a8bc-428a-993d-b3418c60b000-sys\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.671939 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.671845 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d001a36-a8bc-428a-993d-b3418c60b000-tmp\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.671939 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.671893 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d001a36-a8bc-428a-993d-b3418c60b000-etc-tuned\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.672070 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.672027 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ca760ea-272f-4b74-b3ff-958d46cffced-ovn-node-metrics-cert\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.677558 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:39.677536 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:24:39.677787 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:39.677564 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:24:39.677787 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:39.677578 2573 projected.go:196] Error preparing data for projected volume kube-api-access-v77pt for pod openshift-network-diagnostics/network-check-target-pvv46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:24:39.677787 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:39.677632 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt podName:b2859ee6-181c-4e9c-a3f6-777df52447fa nodeName:}" failed. No retries permitted until 2026-07-17 17:24:40.17761558 +0000 UTC m=+1.984396723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v77pt" (UniqueName: "kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt") pod "network-check-target-pvv46" (UID: "b2859ee6-181c-4e9c-a3f6-777df52447fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:24:39.678600 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.678569 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwh42\" (UniqueName: \"kubernetes.io/projected/e97089fa-7036-4cf6-a506-e292a28960d9-kube-api-access-mwh42\") pod \"iptables-alerter-s89r6\" (UID: \"e97089fa-7036-4cf6-a506-e292a28960d9\") " pod="openshift-network-operator/iptables-alerter-s89r6" Jul 17 17:24:39.678709 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.678686 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vg9t\" (UniqueName: \"kubernetes.io/projected/8edbc735-6e23-4f6b-8aaa-4bb0d95655b4-kube-api-access-2vg9t\") pod \"multus-dqkbz\" (UID: \"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4\") " pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.678769 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.678737 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9vnn\" (UniqueName: \"kubernetes.io/projected/b905e39a-a951-4f8a-9479-348a1fea69e0-kube-api-access-q9vnn\") pod \"node-ca-mmd7j\" (UID: \"b905e39a-a951-4f8a-9479-348a1fea69e0\") " pod="openshift-image-registry/node-ca-mmd7j" Jul 17 17:24:39.678960 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.678934 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzkcb\" (UniqueName: \"kubernetes.io/projected/c0ca305d-1ff8-4cab-bde6-bc63609387aa-kube-api-access-nzkcb\") pod \"network-metrics-daemon-w49zh\" (UID: \"c0ca305d-1ff8-4cab-bde6-bc63609387aa\") " pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:39.679891 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.679872 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pctx5\" (UniqueName: \"kubernetes.io/projected/e92dd775-d49f-47b9-bf80-448ae55bb6f2-kube-api-access-pctx5\") pod \"multus-additional-cni-plugins-wjj97\" (UID: \"e92dd775-d49f-47b9-bf80-448ae55bb6f2\") " pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.680312 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.680293 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwnqd\" (UniqueName: \"kubernetes.io/projected/6ca760ea-272f-4b74-b3ff-958d46cffced-kube-api-access-gwnqd\") pod \"ovnkube-node-7lgg2\" (UID: \"6ca760ea-272f-4b74-b3ff-958d46cffced\") " pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.680529 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.680514 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsfqg\" (UniqueName: \"kubernetes.io/projected/4d001a36-a8bc-428a-993d-b3418c60b000-kube-api-access-hsfqg\") pod \"tuned-flc96\" (UID: \"4d001a36-a8bc-428a-993d-b3418c60b000\") " pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.685611 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.685594 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:24:39.716405 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:39.716371 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4336710fed4961d5d53b67aadb1cb02.slice/crio-6833bb0604b7bab2c9a658b7191b828a3576b9027144d6fda8f5295ad639c141 WatchSource:0}: Error finding container 6833bb0604b7bab2c9a658b7191b828a3576b9027144d6fda8f5295ad639c141: Status 404 returned error can't find the container with id 6833bb0604b7bab2c9a658b7191b828a3576b9027144d6fda8f5295ad639c141 Jul 17 17:24:39.716817 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:39.716796 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2d2ffede5ac360572d29fe6cd09b1b.slice/crio-50d9210fda943729366fce6bb197c05c5bfe44e7411ee004baf5a556766ec877 WatchSource:0}: Error finding container 50d9210fda943729366fce6bb197c05c5bfe44e7411ee004baf5a556766ec877: Status 404 returned error can't find the container with id 50d9210fda943729366fce6bb197c05c5bfe44e7411ee004baf5a556766ec877 Jul 17 17:24:39.720715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.720694 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 17:24:39.767879 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.767854 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-sys-fs\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.767879 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.767883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.768125 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.767899 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvd86\" (UniqueName: \"kubernetes.io/projected/e02547e4-091d-40e4-9ed5-321f22875661-kube-api-access-tvd86\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.768125 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.767917 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17bd4e15-f627-4ce8-8992-9e4e0a754e32-tmp-dir\") pod \"node-resolver-9bwpz\" (UID: \"17bd4e15-f627-4ce8-8992-9e4e0a754e32\") " pod="openshift-dns/node-resolver-9bwpz" Jul 17 17:24:39.768125 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.767931 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-dbus\") pod \"global-pull-secret-syncer-hr9d5\" (UID: \"2a96d8c8-87eb-47c6-b427-0486a1b6eb23\") " pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:39.768125 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.767986 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-sys-fs\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.768125 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768000 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-dbus\") pod \"global-pull-secret-syncer-hr9d5\" (UID: \"2a96d8c8-87eb-47c6-b427-0486a1b6eb23\") " pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:39.768125 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.767992 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.768125 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768014 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p8v8\" (UniqueName: \"kubernetes.io/projected/17bd4e15-f627-4ce8-8992-9e4e0a754e32-kube-api-access-6p8v8\") pod \"node-resolver-9bwpz\" (UID: \"17bd4e15-f627-4ce8-8992-9e4e0a754e32\") " pod="openshift-dns/node-resolver-9bwpz" Jul 17 17:24:39.768125 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-registration-dir\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.768125 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/747c8133-308b-4cf0-99cd-d692c4a248ad-agent-certs\") pod \"konnectivity-agent-bk7mr\" (UID: \"747c8133-308b-4cf0-99cd-d692c4a248ad\") " pod="kube-system/konnectivity-agent-bk7mr" Jul 17 17:24:39.768125 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-device-dir\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.768562 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-kubelet-config\") pod \"global-pull-secret-syncer-hr9d5\" (UID: \"2a96d8c8-87eb-47c6-b427-0486a1b6eb23\") " pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:39.768562 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768186 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret\") pod \"global-pull-secret-syncer-hr9d5\" (UID: \"2a96d8c8-87eb-47c6-b427-0486a1b6eb23\") " pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:39.768562 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768210 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17bd4e15-f627-4ce8-8992-9e4e0a754e32-hosts-file\") pod \"node-resolver-9bwpz\" (UID: \"17bd4e15-f627-4ce8-8992-9e4e0a754e32\") " pod="openshift-dns/node-resolver-9bwpz" Jul 17 17:24:39.768562 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768242 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-kubelet-config\") pod \"global-pull-secret-syncer-hr9d5\" (UID: \"2a96d8c8-87eb-47c6-b427-0486a1b6eb23\") " pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:39.768562 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768280 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-device-dir\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.768562 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:39.768294 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:24:39.768562 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768311 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-registration-dir\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.768562 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768339 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17bd4e15-f627-4ce8-8992-9e4e0a754e32-hosts-file\") pod \"node-resolver-9bwpz\" (UID: \"17bd4e15-f627-4ce8-8992-9e4e0a754e32\") " pod="openshift-dns/node-resolver-9bwpz" Jul 17 17:24:39.768562 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768315 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17bd4e15-f627-4ce8-8992-9e4e0a754e32-tmp-dir\") pod \"node-resolver-9bwpz\" (UID: \"17bd4e15-f627-4ce8-8992-9e4e0a754e32\") " pod="openshift-dns/node-resolver-9bwpz" Jul 17 17:24:39.768562 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:39.768399 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret podName:2a96d8c8-87eb-47c6-b427-0486a1b6eb23 nodeName:}" failed. No retries permitted until 2026-07-17 17:24:40.268381467 +0000 UTC m=+2.075162597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret") pod "global-pull-secret-syncer-hr9d5" (UID: "2a96d8c8-87eb-47c6-b427-0486a1b6eb23") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:24:39.768562 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768296 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/747c8133-308b-4cf0-99cd-d692c4a248ad-konnectivity-ca\") pod \"konnectivity-agent-bk7mr\" (UID: \"747c8133-308b-4cf0-99cd-d692c4a248ad\") " pod="kube-system/konnectivity-agent-bk7mr" Jul 17 17:24:39.768562 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768442 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-etc-selinux\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.768562 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768467 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-socket-dir\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.769079 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768590 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-etc-selinux\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.769079 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768598 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e02547e4-091d-40e4-9ed5-321f22875661-socket-dir\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.769079 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.768725 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/747c8133-308b-4cf0-99cd-d692c4a248ad-konnectivity-ca\") pod \"konnectivity-agent-bk7mr\" (UID: \"747c8133-308b-4cf0-99cd-d692c4a248ad\") " pod="kube-system/konnectivity-agent-bk7mr" Jul 17 17:24:39.770455 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.770438 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/747c8133-308b-4cf0-99cd-d692c4a248ad-agent-certs\") pod \"konnectivity-agent-bk7mr\" (UID: \"747c8133-308b-4cf0-99cd-d692c4a248ad\") " pod="kube-system/konnectivity-agent-bk7mr" Jul 17 17:24:39.778582 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.778561 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p8v8\" (UniqueName: \"kubernetes.io/projected/17bd4e15-f627-4ce8-8992-9e4e0a754e32-kube-api-access-6p8v8\") pod \"node-resolver-9bwpz\" (UID: \"17bd4e15-f627-4ce8-8992-9e4e0a754e32\") " pod="openshift-dns/node-resolver-9bwpz" Jul 17 17:24:39.779329 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.779312 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvd86\" (UniqueName: \"kubernetes.io/projected/e02547e4-091d-40e4-9ed5-321f22875661-kube-api-access-tvd86\") pod \"aws-ebs-csi-driver-node-dv7zx\" (UID: \"e02547e4-091d-40e4-9ed5-321f22875661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.832648 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.832626 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:24:39.876923 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.876857 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wjj97" Jul 17 17:24:39.885108 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:39.885087 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92dd775_d49f_47b9_bf80_448ae55bb6f2.slice/crio-82ea36c35bafef1dd4a428db655072184b26728ea79d909f02e5a2e2f83d23c2 WatchSource:0}: Error finding container 82ea36c35bafef1dd4a428db655072184b26728ea79d909f02e5a2e2f83d23c2: Status 404 returned error can't find the container with id 82ea36c35bafef1dd4a428db655072184b26728ea79d909f02e5a2e2f83d23c2 Jul 17 17:24:39.893994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.893976 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dqkbz" Jul 17 17:24:39.900814 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:39.900795 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edbc735_6e23_4f6b_8aaa_4bb0d95655b4.slice/crio-b6ca994cb848b9da0b433a152e3cadaba08fc4c2feaae251ad5845f24c757910 WatchSource:0}: Error finding container b6ca994cb848b9da0b433a152e3cadaba08fc4c2feaae251ad5845f24c757910: Status 404 returned error can't find the container with id b6ca994cb848b9da0b433a152e3cadaba08fc4c2feaae251ad5845f24c757910 Jul 17 17:24:39.906336 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.906318 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s89r6" Jul 17 17:24:39.914436 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:39.914405 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97089fa_7036_4cf6_a506_e292a28960d9.slice/crio-bd172848647464c98595f291adc7886dc766afccd9d110d7c3ad6449982779a5 WatchSource:0}: Error finding container bd172848647464c98595f291adc7886dc766afccd9d110d7c3ad6449982779a5: Status 404 returned error can't find the container with id bd172848647464c98595f291adc7886dc766afccd9d110d7c3ad6449982779a5 Jul 17 17:24:39.927577 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.927559 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:24:39.934736 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:39.934716 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca760ea_272f_4b74_b3ff_958d46cffced.slice/crio-4f71a6bf362e088fc32baa9ef6aa7968ffcdca0a28c548a8c402ba6a7afe1818 WatchSource:0}: Error finding container 4f71a6bf362e088fc32baa9ef6aa7968ffcdca0a28c548a8c402ba6a7afe1818: Status 404 returned error can't find the container with id 4f71a6bf362e088fc32baa9ef6aa7968ffcdca0a28c548a8c402ba6a7afe1818 Jul 17 17:24:39.942248 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.942230 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-flc96" Jul 17 17:24:39.948114 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.948098 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mmd7j" Jul 17 17:24:39.948971 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:39.948951 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d001a36_a8bc_428a_993d_b3418c60b000.slice/crio-6ae5abf8581fc1810b849114c5c07634563bd6ee91c5cfd7b5d81fc09aee60d1 WatchSource:0}: Error finding container 6ae5abf8581fc1810b849114c5c07634563bd6ee91c5cfd7b5d81fc09aee60d1: Status 404 returned error can't find the container with id 6ae5abf8581fc1810b849114c5c07634563bd6ee91c5cfd7b5d81fc09aee60d1 Jul 17 17:24:39.955031 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:39.955012 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb905e39a_a951_4f8a_9479_348a1fea69e0.slice/crio-e6e848be78ed98a85a7de958cbb758a59dbc1f7b3f1f9e3af32e86261b811bf8 WatchSource:0}: Error finding container e6e848be78ed98a85a7de958cbb758a59dbc1f7b3f1f9e3af32e86261b811bf8: Status 404 returned error can't find the container with id e6e848be78ed98a85a7de958cbb758a59dbc1f7b3f1f9e3af32e86261b811bf8 Jul 17 17:24:39.967856 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.967838 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bk7mr" Jul 17 17:24:39.974946 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:39.974927 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747c8133_308b_4cf0_99cd_d692c4a248ad.slice/crio-70cdb5f7ef421160f093ebeb9f74f25d5585334837cdef6561e68690b1e98d62 WatchSource:0}: Error finding container 70cdb5f7ef421160f093ebeb9f74f25d5585334837cdef6561e68690b1e98d62: Status 404 returned error can't find the container with id 70cdb5f7ef421160f093ebeb9f74f25d5585334837cdef6561e68690b1e98d62 Jul 17 17:24:39.981808 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.981789 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" Jul 17 17:24:39.985835 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:39.985667 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9bwpz" Jul 17 17:24:39.992565 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:39.992545 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02547e4_091d_40e4_9ed5_321f22875661.slice/crio-ab5a47e96076967cf3bee53e03f7ab8c680fa55cb67acdfbb39a2c9872fdd48e WatchSource:0}: Error finding container ab5a47e96076967cf3bee53e03f7ab8c680fa55cb67acdfbb39a2c9872fdd48e: Status 404 returned error can't find the container with id ab5a47e96076967cf3bee53e03f7ab8c680fa55cb67acdfbb39a2c9872fdd48e Jul 17 17:24:39.994580 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:24:39.994555 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bd4e15_f627_4ce8_8992_9e4e0a754e32.slice/crio-29b9a254231e1fec8a08f62e9c750ce0b673e3121bea155e90f8639cb4bd21b8 WatchSource:0}: Error finding container 29b9a254231e1fec8a08f62e9c750ce0b673e3121bea155e90f8639cb4bd21b8: Status 404 returned error can't find the container with id 29b9a254231e1fec8a08f62e9c750ce0b673e3121bea155e90f8639cb4bd21b8 Jul 17 17:24:40.170399 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:40.170359 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs\") pod \"network-metrics-daemon-w49zh\" (UID: \"c0ca305d-1ff8-4cab-bde6-bc63609387aa\") " pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:40.170576 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:40.170495 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:24:40.170576 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:40.170558 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs podName:c0ca305d-1ff8-4cab-bde6-bc63609387aa nodeName:}" failed. No retries permitted until 2026-07-17 17:24:41.170539027 +0000 UTC m=+2.977320153 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs") pod "network-metrics-daemon-w49zh" (UID: "c0ca305d-1ff8-4cab-bde6-bc63609387aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:24:41.342975 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:41.342691 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret\") pod \"global-pull-secret-syncer-hr9d5\" (UID: \"2a96d8c8-87eb-47c6-b427-0486a1b6eb23\") " pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:41.342975 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:41.342743 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77pt\" (UniqueName: \"kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt\") pod \"network-check-target-pvv46\" (UID: \"b2859ee6-181c-4e9c-a3f6-777df52447fa\") " pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:41.342975 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:41.342854 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:24:41.342975 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:41.342870 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:24:41.342975 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:41.342889 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:24:41.342975 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:41.342902 2573 projected.go:196] Error preparing data for projected volume kube-api-access-v77pt for pod openshift-network-diagnostics/network-check-target-pvv46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:24:41.342975 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:41.342927 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret podName:2a96d8c8-87eb-47c6-b427-0486a1b6eb23 nodeName:}" failed. No retries permitted until 2026-07-17 17:24:42.342904832 +0000 UTC m=+3.078328872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret") pod "global-pull-secret-syncer-hr9d5" (UID: "2a96d8c8-87eb-47c6-b427-0486a1b6eb23") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:24:41.342975 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:41.342950 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt podName:b2859ee6-181c-4e9c-a3f6-777df52447fa nodeName:}" failed. No retries permitted until 2026-07-17 17:24:42.342934804 +0000 UTC m=+3.078358862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v77pt" (UniqueName: "kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt") pod "network-check-target-pvv46" (UID: "b2859ee6-181c-4e9c-a3f6-777df52447fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:24:41.697166 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:41.694635 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 17:19:40 +0000 UTC" deadline="2028-01-10 18:23:27.706837455 +0000 UTC" Jul 17 17:24:41.697166 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:41.694696 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13008h58m46.012145059s" Jul 17 17:24:41.816177 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:41.816062 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9bwpz" event={"ID":"17bd4e15-f627-4ce8-8992-9e4e0a754e32","Type":"ContainerStarted","Data":"29b9a254231e1fec8a08f62e9c750ce0b673e3121bea155e90f8639cb4bd21b8"} Jul 17 17:24:41.844171 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:41.844127 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bk7mr" event={"ID":"747c8133-308b-4cf0-99cd-d692c4a248ad","Type":"ContainerStarted","Data":"70cdb5f7ef421160f093ebeb9f74f25d5585334837cdef6561e68690b1e98d62"} Jul 17 17:24:41.857487 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:41.857408 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-flc96" event={"ID":"4d001a36-a8bc-428a-993d-b3418c60b000","Type":"ContainerStarted","Data":"6ae5abf8581fc1810b849114c5c07634563bd6ee91c5cfd7b5d81fc09aee60d1"} Jul 17 17:24:41.883807 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:41.883772 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s89r6" event={"ID":"e97089fa-7036-4cf6-a506-e292a28960d9","Type":"ContainerStarted","Data":"bd172848647464c98595f291adc7886dc766afccd9d110d7c3ad6449982779a5"} Jul 17 17:24:41.903434 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:41.903392 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dqkbz" event={"ID":"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4","Type":"ContainerStarted","Data":"b6ca994cb848b9da0b433a152e3cadaba08fc4c2feaae251ad5845f24c757910"} Jul 17 17:24:41.930579 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:41.930540 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjj97" event={"ID":"e92dd775-d49f-47b9-bf80-448ae55bb6f2","Type":"ContainerStarted","Data":"82ea36c35bafef1dd4a428db655072184b26728ea79d909f02e5a2e2f83d23c2"} Jul 17 17:24:41.951855 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:41.951741 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" event={"ID":"3e2d2ffede5ac360572d29fe6cd09b1b","Type":"ContainerStarted","Data":"50d9210fda943729366fce6bb197c05c5bfe44e7411ee004baf5a556766ec877"} Jul 17 17:24:41.965203 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:41.965117 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" event={"ID":"e02547e4-091d-40e4-9ed5-321f22875661","Type":"ContainerStarted","Data":"ab5a47e96076967cf3bee53e03f7ab8c680fa55cb67acdfbb39a2c9872fdd48e"} Jul 17 17:24:41.977240 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:41.977202 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mmd7j" event={"ID":"b905e39a-a951-4f8a-9479-348a1fea69e0","Type":"ContainerStarted","Data":"e6e848be78ed98a85a7de958cbb758a59dbc1f7b3f1f9e3af32e86261b811bf8"} Jul 17 17:24:42.000885 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:42.000782 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" event={"ID":"6ca760ea-272f-4b74-b3ff-958d46cffced","Type":"ContainerStarted","Data":"4f71a6bf362e088fc32baa9ef6aa7968ffcdca0a28c548a8c402ba6a7afe1818"} Jul 17 17:24:42.004773 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:42.004742 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" event={"ID":"c4336710fed4961d5d53b67aadb1cb02","Type":"ContainerStarted","Data":"6833bb0604b7bab2c9a658b7191b828a3576b9027144d6fda8f5295ad639c141"} Jul 17 17:24:42.126730 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:42.126699 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:24:42.251442 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:42.251402 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs\") pod \"network-metrics-daemon-w49zh\" (UID: \"c0ca305d-1ff8-4cab-bde6-bc63609387aa\") " pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:42.251652 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:42.251598 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:24:42.251726 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:42.251681 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs podName:c0ca305d-1ff8-4cab-bde6-bc63609387aa nodeName:}" failed. No retries permitted until 2026-07-17 17:24:44.251660274 +0000 UTC m=+4.987084325 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs") pod "network-metrics-daemon-w49zh" (UID: "c0ca305d-1ff8-4cab-bde6-bc63609387aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:24:42.352188 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:42.352092 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret\") pod \"global-pull-secret-syncer-hr9d5\" (UID: \"2a96d8c8-87eb-47c6-b427-0486a1b6eb23\") " pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:42.352188 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:42.352146 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77pt\" (UniqueName: \"kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt\") pod \"network-check-target-pvv46\" (UID: \"b2859ee6-181c-4e9c-a3f6-777df52447fa\") " pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:42.352461 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:42.352311 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:24:42.352461 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:42.352332 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:24:42.352461 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:42.352344 2573 projected.go:196] Error preparing data for projected volume kube-api-access-v77pt for pod openshift-network-diagnostics/network-check-target-pvv46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:24:42.352461 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:42.352397 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt podName:b2859ee6-181c-4e9c-a3f6-777df52447fa nodeName:}" failed. No retries permitted until 2026-07-17 17:24:44.352379576 +0000 UTC m=+5.087803617 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v77pt" (UniqueName: "kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt") pod "network-check-target-pvv46" (UID: "b2859ee6-181c-4e9c-a3f6-777df52447fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:24:42.353136 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:42.352856 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:24:42.353136 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:42.352915 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret podName:2a96d8c8-87eb-47c6-b427-0486a1b6eb23 nodeName:}" failed. No retries permitted until 2026-07-17 17:24:44.352896676 +0000 UTC m=+5.088320717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret") pod "global-pull-secret-syncer-hr9d5" (UID: "2a96d8c8-87eb-47c6-b427-0486a1b6eb23") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:24:42.695217 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:42.695090 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 17:19:40 +0000 UTC" deadline="2028-02-17 12:09:30.994327241 +0000 UTC" Jul 17 17:24:42.695217 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:42.695170 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13914h44m48.299162152s" Jul 17 17:24:42.771082 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:42.771033 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:42.771566 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:42.771150 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:24:42.771566 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:42.771543 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:42.771877 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:42.771670 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:24:42.771877 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:42.771742 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:42.771877 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:42.771806 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:24:44.268507 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:44.268428 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs\") pod \"network-metrics-daemon-w49zh\" (UID: \"c0ca305d-1ff8-4cab-bde6-bc63609387aa\") " pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:44.269000 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:44.268603 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:24:44.269000 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:44.268693 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs podName:c0ca305d-1ff8-4cab-bde6-bc63609387aa nodeName:}" failed. No retries permitted until 2026-07-17 17:24:48.268673505 +0000 UTC m=+9.004097547 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs") pod "network-metrics-daemon-w49zh" (UID: "c0ca305d-1ff8-4cab-bde6-bc63609387aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:24:44.370158 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:44.370124 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret\") pod \"global-pull-secret-syncer-hr9d5\" (UID: \"2a96d8c8-87eb-47c6-b427-0486a1b6eb23\") " pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:44.370332 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:44.370181 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77pt\" (UniqueName: \"kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt\") pod \"network-check-target-pvv46\" (UID: \"b2859ee6-181c-4e9c-a3f6-777df52447fa\") " pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:44.370398 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:44.370355 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:24:44.370398 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:44.370374 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:24:44.370398 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:44.370388 2573 projected.go:196] Error preparing data for projected volume kube-api-access-v77pt for pod openshift-network-diagnostics/network-check-target-pvv46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:24:44.370534 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:44.370444 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt podName:b2859ee6-181c-4e9c-a3f6-777df52447fa nodeName:}" failed. No retries permitted until 2026-07-17 17:24:48.370425722 +0000 UTC m=+9.105849760 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v77pt" (UniqueName: "kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt") pod "network-check-target-pvv46" (UID: "b2859ee6-181c-4e9c-a3f6-777df52447fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:24:44.370943 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:44.370833 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:24:44.370943 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:44.370884 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret podName:2a96d8c8-87eb-47c6-b427-0486a1b6eb23 nodeName:}" failed. No retries permitted until 2026-07-17 17:24:48.370870457 +0000 UTC m=+9.106294487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret") pod "global-pull-secret-syncer-hr9d5" (UID: "2a96d8c8-87eb-47c6-b427-0486a1b6eb23") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:24:44.772329 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:44.771875 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:44.772329 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:44.771915 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:44.772329 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:44.771913 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:44.772329 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:44.771998 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:24:44.772329 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:44.772120 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:24:44.772329 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:44.772208 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:24:46.771785 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:46.771258 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:46.771785 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:46.771293 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:46.771785 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:46.771297 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:46.771785 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:46.771398 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:24:46.771785 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:46.771497 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:24:46.771785 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:46.771583 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:24:48.305007 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:48.304965 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs\") pod \"network-metrics-daemon-w49zh\" (UID: \"c0ca305d-1ff8-4cab-bde6-bc63609387aa\") " pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:48.305459 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:48.305118 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:24:48.305459 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:48.305190 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs podName:c0ca305d-1ff8-4cab-bde6-bc63609387aa nodeName:}" failed. No retries permitted until 2026-07-17 17:24:56.305170483 +0000 UTC m=+17.040594737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs") pod "network-metrics-daemon-w49zh" (UID: "c0ca305d-1ff8-4cab-bde6-bc63609387aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:24:48.405710 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:48.405666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret\") pod \"global-pull-secret-syncer-hr9d5\" (UID: \"2a96d8c8-87eb-47c6-b427-0486a1b6eb23\") " pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:48.405887 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:48.405716 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77pt\" (UniqueName: \"kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt\") pod \"network-check-target-pvv46\" (UID: \"b2859ee6-181c-4e9c-a3f6-777df52447fa\") " pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:48.405887 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:48.405844 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:24:48.406012 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:48.405918 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret podName:2a96d8c8-87eb-47c6-b427-0486a1b6eb23 nodeName:}" failed. No retries permitted until 2026-07-17 17:24:56.40589614 +0000 UTC m=+17.141320181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret") pod "global-pull-secret-syncer-hr9d5" (UID: "2a96d8c8-87eb-47c6-b427-0486a1b6eb23") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:24:48.406012 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:48.405945 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:24:48.406012 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:48.405966 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:24:48.406012 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:48.405979 2573 projected.go:196] Error preparing data for projected volume kube-api-access-v77pt for pod openshift-network-diagnostics/network-check-target-pvv46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:24:48.406227 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:48.406111 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt podName:b2859ee6-181c-4e9c-a3f6-777df52447fa nodeName:}" failed. No retries permitted until 2026-07-17 17:24:56.406075962 +0000 UTC m=+17.141500006 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v77pt" (UniqueName: "kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt") pod "network-check-target-pvv46" (UID: "b2859ee6-181c-4e9c-a3f6-777df52447fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:24:48.771840 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:48.771542 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:48.771840 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:48.771693 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:48.771840 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:48.771720 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:48.771840 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:48.771686 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:24:48.771840 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:48.771784 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:24:48.772203 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:48.771876 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:24:50.771544 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:50.771507 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:50.772015 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:50.771650 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:24:50.772085 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:50.772060 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:50.772331 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:50.772166 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:24:50.772331 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:50.772231 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:50.772331 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:50.772289 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:24:52.772267 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:52.772023 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:52.772755 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:52.772023 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:52.772755 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:52.772377 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:24:52.772755 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:52.772039 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:52.772755 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:52.772475 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:24:52.772755 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:52.772570 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:24:54.771633 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:54.771582 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:54.771633 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:54.771609 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:54.772119 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:54.771613 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:54.772119 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:54.771741 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:24:54.772119 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:54.771811 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:24:54.772119 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:54.771901 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:24:56.364057 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:56.364016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs\") pod \"network-metrics-daemon-w49zh\" (UID: \"c0ca305d-1ff8-4cab-bde6-bc63609387aa\") " pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:56.364549 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:56.364203 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:24:56.364549 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:56.364287 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs podName:c0ca305d-1ff8-4cab-bde6-bc63609387aa nodeName:}" failed. No retries permitted until 2026-07-17 17:25:12.364265782 +0000 UTC m=+33.099689816 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs") pod "network-metrics-daemon-w49zh" (UID: "c0ca305d-1ff8-4cab-bde6-bc63609387aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:24:56.465167 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:56.465120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret\") pod \"global-pull-secret-syncer-hr9d5\" (UID: \"2a96d8c8-87eb-47c6-b427-0486a1b6eb23\") " pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:56.465167 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:56.465168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77pt\" (UniqueName: \"kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt\") pod \"network-check-target-pvv46\" (UID: \"b2859ee6-181c-4e9c-a3f6-777df52447fa\") " pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:56.465428 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:56.465272 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:24:56.465428 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:56.465289 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:24:56.465428 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:56.465311 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:24:56.465428 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:56.465324 2573 projected.go:196] Error preparing data for projected volume kube-api-access-v77pt for pod openshift-network-diagnostics/network-check-target-pvv46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:24:56.465428 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:56.465352 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret podName:2a96d8c8-87eb-47c6-b427-0486a1b6eb23 nodeName:}" failed. No retries permitted until 2026-07-17 17:25:12.465332288 +0000 UTC m=+33.200756315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret") pod "global-pull-secret-syncer-hr9d5" (UID: "2a96d8c8-87eb-47c6-b427-0486a1b6eb23") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:24:56.465428 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:56.465373 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt podName:b2859ee6-181c-4e9c-a3f6-777df52447fa nodeName:}" failed. No retries permitted until 2026-07-17 17:25:12.465363002 +0000 UTC m=+33.200787033 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v77pt" (UniqueName: "kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt") pod "network-check-target-pvv46" (UID: "b2859ee6-181c-4e9c-a3f6-777df52447fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:24:56.772039 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:56.772008 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:56.772039 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:56.772026 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:56.772263 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:56.772032 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:56.772263 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:56.772129 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:24:56.772263 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:56.772232 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:24:56.772403 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:56.772336 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:24:58.771363 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:58.771328 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:24:58.771919 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:58.771334 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:24:58.771919 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:58.771452 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:24:58.771919 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:58.771552 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:24:58.771919 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:24:58.771347 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:24:58.771919 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:24:58.771671 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:25:00.771611 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:00.771587 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:00.771918 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:00.771690 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:00.771918 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:00.771698 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:25:00.771918 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:00.771718 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:00.771918 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:00.771788 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:25:00.771918 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:00.771840 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:25:01.042630 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:01.042588 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-flc96" event={"ID":"4d001a36-a8bc-428a-993d-b3418c60b000","Type":"ContainerStarted","Data":"97ca89ae0209ee37cd9f1dd6b98fda077a8d98c989b560de3ab163a33f6981dc"} Jul 17 17:25:01.044037 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:01.044011 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" event={"ID":"3e2d2ffede5ac360572d29fe6cd09b1b","Type":"ContainerStarted","Data":"bf815965e5c058f80beccc395f6237717fa88b83d27e674273fb5a2780f3b716"} Jul 17 17:25:01.045433 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:01.045413 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" event={"ID":"6ca760ea-272f-4b74-b3ff-958d46cffced","Type":"ContainerStarted","Data":"4ae1182cd55d8555271517ed3e7f624281a7cba500fd5803a7436c7d6068b497"} Jul 17 17:25:01.076330 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:01.076276 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-183.ec2.internal" podStartSLOduration=21.076257509 podStartE2EDuration="21.076257509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:01.075598587 +0000 UTC m=+21.811022660" watchObservedRunningTime="2026-07-17 17:25:01.076257509 +0000 UTC m=+21.811681561" Jul 17 17:25:01.076555 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:01.076527 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-flc96" podStartSLOduration=2.397583628 podStartE2EDuration="22.076516963s" totalImagesPullingTime="19.678933335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:01.06135227 +0000 UTC m=+21.796776334" watchObservedRunningTime="2026-07-17 17:25:01.076516963 +0000 UTC m=+21.811941014" Jul 17 17:25:02.048552 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.048182 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9bwpz" event={"ID":"17bd4e15-f627-4ce8-8992-9e4e0a754e32","Type":"ContainerStarted","Data":"294cc28f991d2e81c210202135a5c886b9f1198a8ad76a84f0b3518bfb7daa3c"} Jul 17 17:25:02.049639 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.049600 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bk7mr" event={"ID":"747c8133-308b-4cf0-99cd-d692c4a248ad","Type":"ContainerStarted","Data":"273d7e64beea0e0e66b0227d61afae84f3a219f565f15569042c99838b6db396"} Jul 17 17:25:02.050837 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.050818 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s89r6" event={"ID":"e97089fa-7036-4cf6-a506-e292a28960d9","Type":"ContainerStarted","Data":"1535ab67189e3d90232b8b29508ed242c1bbe21f06ceb4c9d956ec2389a94318"} Jul 17 17:25:02.052083 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.052060 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dqkbz" event={"ID":"8edbc735-6e23-4f6b-8aaa-4bb0d95655b4","Type":"ContainerStarted","Data":"029019cc9dbe88931a470d649cfcddfa5540ce603642b3d87941c43d242b205d"} Jul 17 17:25:02.053340 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.053320 2573 generic.go:356] "Generic (PLEG): container finished" podID="e92dd775-d49f-47b9-bf80-448ae55bb6f2" containerID="7ed41056e9891e6f95d287320a77d6903aa7002e66fe2086f32434f2b9727050" exitCode=0 Jul 17 17:25:02.053504 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.053378 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjj97" event={"ID":"e92dd775-d49f-47b9-bf80-448ae55bb6f2","Type":"ContainerDied","Data":"7ed41056e9891e6f95d287320a77d6903aa7002e66fe2086f32434f2b9727050"} Jul 17 17:25:02.054762 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.054714 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" event={"ID":"e02547e4-091d-40e4-9ed5-321f22875661","Type":"ContainerStarted","Data":"511690b3195173e582273652f0e153a10ed6769f6cf3f3bceca977a211e77c0f"} Jul 17 17:25:02.056027 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.056008 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mmd7j" event={"ID":"b905e39a-a951-4f8a-9479-348a1fea69e0","Type":"ContainerStarted","Data":"401ce125d05139833fb7b9d3298665b88586d77091fec53775f925eb6caa9f04"} Jul 17 17:25:02.058692 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.058674 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:25:02.059033 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.059010 2573 generic.go:356] "Generic (PLEG): container finished" podID="6ca760ea-272f-4b74-b3ff-958d46cffced" containerID="3d9e71ceee6081ee839513bc9b033c5b043d7997e8f7d273d5102162389a97e1" exitCode=1 Jul 17 17:25:02.059106 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.059086 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" event={"ID":"6ca760ea-272f-4b74-b3ff-958d46cffced","Type":"ContainerStarted","Data":"4a7ce2a8e610f5f7108ceae858a2aef405da960874570b4156e3924e7f39baa6"} Jul 17 17:25:02.059197 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.059182 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" event={"ID":"6ca760ea-272f-4b74-b3ff-958d46cffced","Type":"ContainerStarted","Data":"cf01b8c6491f30420af3f5904ababbe4e575d725583e3c32759058d0bd91a0f5"} Jul 17 17:25:02.059250 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.059238 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" event={"ID":"6ca760ea-272f-4b74-b3ff-958d46cffced","Type":"ContainerStarted","Data":"0b2b9fc9af68998d81cddf95641ddaae52f21a8ab12c6995f09c4836a2991753"} Jul 17 17:25:02.059303 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.059285 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" event={"ID":"6ca760ea-272f-4b74-b3ff-958d46cffced","Type":"ContainerStarted","Data":"8cae15f4c08bd5abf0bcb874822bd6ea061ba11af8e4c7b842b80868e2d66348"} Jul 17 17:25:02.059359 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.059332 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" event={"ID":"6ca760ea-272f-4b74-b3ff-958d46cffced","Type":"ContainerDied","Data":"3d9e71ceee6081ee839513bc9b033c5b043d7997e8f7d273d5102162389a97e1"} Jul 17 17:25:02.060843 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.060824 2573 generic.go:356] "Generic (PLEG): container finished" podID="c4336710fed4961d5d53b67aadb1cb02" containerID="3d30ec83a284e273fb0d8186b88a6f8b6644250c5919723b601eb63d56138811" exitCode=0 Jul 17 17:25:02.060902 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.060860 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" event={"ID":"c4336710fed4961d5d53b67aadb1cb02","Type":"ContainerDied","Data":"3d30ec83a284e273fb0d8186b88a6f8b6644250c5919723b601eb63d56138811"} Jul 17 17:25:02.067801 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.067764 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9bwpz" podStartSLOduration=2.434516995 podStartE2EDuration="22.067752635s" totalImagesPullingTime="19.63323564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:02.0672266 +0000 UTC m=+22.802650668" watchObservedRunningTime="2026-07-17 17:25:02.067752635 +0000 UTC m=+22.803176684" Jul 17 17:25:02.117833 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.117789 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dqkbz" podStartSLOduration=3.006841963 podStartE2EDuration="23.117772931s" totalImagesPullingTime="20.110930968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:02.103155693 +0000 UTC m=+22.838579742" watchObservedRunningTime="2026-07-17 17:25:02.117772931 +0000 UTC m=+22.853196988" Jul 17 17:25:02.138870 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.138826 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bk7mr" podStartSLOduration=6.54411855 podStartE2EDuration="22.138810038s" totalImagesPullingTime="15.594691488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:02.117647738 +0000 UTC m=+22.853071781" watchObservedRunningTime="2026-07-17 17:25:02.138810038 +0000 UTC m=+22.874234086" Jul 17 17:25:02.153883 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.153839 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mmd7j" podStartSLOduration=3.5052782369999997 podStartE2EDuration="23.153823711s" totalImagesPullingTime="19.648545474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:02.153514348 +0000 UTC m=+22.888938400" watchObservedRunningTime="2026-07-17 17:25:02.153823711 +0000 UTC m=+22.889247759" Jul 17 17:25:02.467020 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.466979 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bk7mr" Jul 17 17:25:02.467676 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.467644 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bk7mr" Jul 17 17:25:02.586245 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.586222 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 17:25:02.695780 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.695674 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T17:25:02.586241881Z","UUID":"a9419bf8-86fd-4bff-b4c5-223ad39a0ca5","Handler":null,"Name":"","Endpoint":""} Jul 17 17:25:02.697332 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.697315 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 17:25:02.697470 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.697353 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 17:25:02.771868 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.771835 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:02.771868 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.771865 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:02.772094 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:02.771961 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:25:02.772094 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:02.771966 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:02.772094 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:02.772065 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:25:02.772255 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:02.772158 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:25:03.065876 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:03.065846 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" event={"ID":"e02547e4-091d-40e4-9ed5-321f22875661","Type":"ContainerStarted","Data":"0514e84363a760814b7a76a35a4a82ffeda22d4770129be756f79c83f0bbe595"} Jul 17 17:25:03.067876 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:03.067849 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" event={"ID":"c4336710fed4961d5d53b67aadb1cb02","Type":"ContainerStarted","Data":"f1b8b74dab1062eace47919c03db5f9d541cc0fcfee3e8089bcc4ad11cab9b36"} Jul 17 17:25:03.068251 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:03.068232 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bk7mr" Jul 17 17:25:03.068533 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:03.068516 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bk7mr" Jul 17 17:25:03.083392 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:03.083351 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-183.ec2.internal" podStartSLOduration=23.083338845 podStartE2EDuration="23.083338845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:03.082943303 +0000 UTC m=+23.818367351" watchObservedRunningTime="2026-07-17 17:25:03.083338845 +0000 UTC m=+23.818762893" Jul 17 17:25:03.113686 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:03.113639 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s89r6" podStartSLOduration=4.398160518 podStartE2EDuration="24.113607291s" totalImagesPullingTime="19.715446773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:03.113113295 +0000 UTC m=+23.848537348" watchObservedRunningTime="2026-07-17 17:25:03.113607291 +0000 UTC m=+23.849031339" Jul 17 17:25:03.350582 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:03.350508 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9bwpz_17bd4e15-f627-4ce8-8992-9e4e0a754e32/dns-node-resolver/0.log" Jul 17 17:25:04.068319 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:04.068267 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:04.074394 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:04.074369 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" event={"ID":"e02547e4-091d-40e4-9ed5-321f22875661","Type":"ContainerStarted","Data":"e901889f54ba9a63f66f04931cbbb5e58cb330b55aa1dab0da4c013cb4adfb3f"} Jul 17 17:25:04.077632 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:04.077598 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:25:04.078074 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:04.078047 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" event={"ID":"6ca760ea-272f-4b74-b3ff-958d46cffced","Type":"ContainerStarted","Data":"8b24b590382a00a82abe257a919cbe06d2747644c0a93b7bd8f06cf6c404017f"} Jul 17 17:25:04.091837 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:04.091794 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv7zx" podStartSLOduration=2.152047312 podStartE2EDuration="24.091781288s" totalImagesPullingTime="21.939733976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:24:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:04.091221343 +0000 UTC m=+24.826645423" watchObservedRunningTime="2026-07-17 17:25:04.091781288 +0000 UTC m=+24.827205334" Jul 17 17:25:04.169799 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:04.169769 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:04.276975 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:04.276950 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:04.370848 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:04.370771 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:04.474669 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:04.474641 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:04.749019 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:04.748996 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mmd7j_b905e39a-a951-4f8a-9479-348a1fea69e0/node-ca/0.log" Jul 17 17:25:04.771324 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:04.771280 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:04.771466 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:04.771281 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:04.771466 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:04.771404 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:25:04.771466 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:04.771291 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:04.771634 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:04.771511 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:25:04.771634 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:04.771577 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:25:04.986308 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:04.986266 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:05.567233 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:05.567186 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:05.973592 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:05.973558 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:06.165501 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:06.165434 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:06.767195 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:06.767170 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:06.771147 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:06.771132 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:06.771240 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:06.771227 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:06.771278 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:06.771239 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:25:06.771327 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:06.771316 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:06.771421 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:06.771403 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:25:06.771461 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:06.771311 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:25:07.088089 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:07.087925 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:25:07.088341 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:07.088310 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" event={"ID":"6ca760ea-272f-4b74-b3ff-958d46cffced","Type":"ContainerStarted","Data":"83013c657cc72e4ee25a65d2db0b82f731bcae9be015e10716dc1d5fc3e7a6bf"} Jul 17 17:25:07.088567 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:07.088536 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:25:07.088677 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:07.088636 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:25:07.088803 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:07.088788 2573 scope.go:117] "RemoveContainer" containerID="3d9e71ceee6081ee839513bc9b033c5b043d7997e8f7d273d5102162389a97e1" Jul 17 17:25:07.103730 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:07.103711 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:25:07.104254 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:07.104199 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:25:07.158778 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:07.158756 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:25:07.166847 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:07.166677 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:07.366039 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:07.365956 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:07.760388 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:07.760355 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:07.813811 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:07.813774 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:08.093674 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:08.093650 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:25:08.094041 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:08.094019 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" event={"ID":"6ca760ea-272f-4b74-b3ff-958d46cffced","Type":"ContainerStarted","Data":"02fe912dbcaed9838e0480308e6006e9775be46eb616837330f651a5ec0aa7ef"} Jul 17 17:25:08.095765 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:08.095741 2573 generic.go:356] "Generic (PLEG): container finished" podID="e92dd775-d49f-47b9-bf80-448ae55bb6f2" containerID="d1793fd3a27310d1a36484f5a19dea7a7494565dc31c580c8b237d456e5c1d78" exitCode=0 Jul 17 17:25:08.095873 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:08.095774 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjj97" event={"ID":"e92dd775-d49f-47b9-bf80-448ae55bb6f2","Type":"ContainerDied","Data":"d1793fd3a27310d1a36484f5a19dea7a7494565dc31c580c8b237d456e5c1d78"} Jul 17 17:25:08.120300 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:08.120251 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" podStartSLOduration=9.407032826 podStartE2EDuration="29.120236433s" totalImagesPullingTime="19.713203607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:08.118985285 +0000 UTC m=+28.854409335" watchObservedRunningTime="2026-07-17 17:25:08.120236433 +0000 UTC m=+28.855660482" Jul 17 17:25:08.197511 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:08.197463 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:08.367418 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:08.367331 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:08.761897 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:08.761868 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:08.771698 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:08.771668 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:08.771840 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:08.771706 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:08.771840 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:08.771794 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:25:08.771955 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:08.771865 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:25:08.771955 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:08.771884 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:08.772056 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:08.771978 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:25:08.862300 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:08.862272 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:09.166438 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:09.166413 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:09.569149 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:09.569121 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:09.767235 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:09.767202 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:10.101715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:10.101682 2573 generic.go:356] "Generic (PLEG): container finished" podID="e92dd775-d49f-47b9-bf80-448ae55bb6f2" containerID="e6bd587c4d9f81b0ed4828dd74717fe3b2e6b4db620567835a85e5f20e68ce60" exitCode=0 Jul 17 17:25:10.102083 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:10.101737 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjj97" event={"ID":"e92dd775-d49f-47b9-bf80-448ae55bb6f2","Type":"ContainerDied","Data":"e6bd587c4d9f81b0ed4828dd74717fe3b2e6b4db620567835a85e5f20e68ce60"} Jul 17 17:25:10.166081 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:10.166052 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:10.407301 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:10.407233 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:10.577545 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:10.577511 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:10.771161 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:10.771132 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:10.771331 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:10.771236 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:10.771331 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:10.771241 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:25:10.771455 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:10.771339 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:25:10.771455 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:10.771242 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:10.771531 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:10.771456 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:25:11.372687 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:11.372602 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:11.422393 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:11.422358 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:11.768386 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:11.768360 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:11.967281 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:11.967245 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:12.108372 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:12.108291 2573 generic.go:356] "Generic (PLEG): container finished" podID="e92dd775-d49f-47b9-bf80-448ae55bb6f2" containerID="4515062ac2ea31de8cd3940cefdc4baf05748cf695752bf2d45b6071d2c7f1c8" exitCode=0 Jul 17 17:25:12.108372 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:12.108348 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjj97" event={"ID":"e92dd775-d49f-47b9-bf80-448ae55bb6f2","Type":"ContainerDied","Data":"4515062ac2ea31de8cd3940cefdc4baf05748cf695752bf2d45b6071d2c7f1c8"} Jul 17 17:25:12.170610 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:12.170577 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:12.366834 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:12.366758 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:12.385819 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:12.385792 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs\") pod \"network-metrics-daemon-w49zh\" (UID: \"c0ca305d-1ff8-4cab-bde6-bc63609387aa\") " pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:12.386126 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:12.385928 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:25:12.386126 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:12.385983 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs podName:c0ca305d-1ff8-4cab-bde6-bc63609387aa nodeName:}" failed. No retries permitted until 2026-07-17 17:25:44.385968725 +0000 UTC m=+65.121392757 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs") pod "network-metrics-daemon-w49zh" (UID: "c0ca305d-1ff8-4cab-bde6-bc63609387aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:25:12.486733 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:12.486698 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret\") pod \"global-pull-secret-syncer-hr9d5\" (UID: \"2a96d8c8-87eb-47c6-b427-0486a1b6eb23\") " pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:12.486895 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:12.486763 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:25:12.486895 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:12.486815 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret podName:2a96d8c8-87eb-47c6-b427-0486a1b6eb23 nodeName:}" failed. No retries permitted until 2026-07-17 17:25:44.486800511 +0000 UTC m=+65.222224539 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret") pod "global-pull-secret-syncer-hr9d5" (UID: "2a96d8c8-87eb-47c6-b427-0486a1b6eb23") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:25:12.486895 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:12.486767 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77pt\" (UniqueName: \"kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt\") pod \"network-check-target-pvv46\" (UID: \"b2859ee6-181c-4e9c-a3f6-777df52447fa\") " pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:12.486895 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:12.486844 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:25:12.486895 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:12.486865 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:25:12.486895 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:12.486877 2573 projected.go:196] Error preparing data for projected volume kube-api-access-v77pt for pod openshift-network-diagnostics/network-check-target-pvv46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:25:12.487159 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:12.486961 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt podName:b2859ee6-181c-4e9c-a3f6-777df52447fa nodeName:}" failed. No retries permitted until 2026-07-17 17:25:44.486948249 +0000 UTC m=+65.222372287 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v77pt" (UniqueName: "kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt") pod "network-check-target-pvv46" (UID: "b2859ee6-181c-4e9c-a3f6-777df52447fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:25:12.771029 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:12.770997 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:12.771216 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:12.770997 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:12.771216 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:12.771117 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:25:12.771216 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:12.770997 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:12.771369 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:12.771186 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:25:12.771369 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:12.771310 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:25:13.380679 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:13.380638 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:13.793678 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:13.793652 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:13.969668 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:13.969637 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:14.173924 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:14.173849 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:14.573051 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:14.573011 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:14.771096 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:14.771067 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:14.771263 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:14.771069 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:14.771263 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:14.771182 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:25:14.771263 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:14.771078 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:14.771263 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:14.771254 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:25:14.771427 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:14.771334 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:25:14.781129 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:14.781106 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:15.245275 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:15.245237 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:16.771421 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:16.771382 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:16.771891 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:16.771382 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:16.771891 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:16.771519 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:25:16.771891 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:16.771392 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:16.771891 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:16.771739 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:25:16.771891 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:16.771614 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:25:17.122533 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:17.122507 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjj97" event={"ID":"e92dd775-d49f-47b9-bf80-448ae55bb6f2","Type":"ContainerStarted","Data":"7b649a522aea98c35ebf35569890e580334eca9ac7d73ce447954cc45dd48828"} Jul 17 17:25:18.126919 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:18.126883 2573 generic.go:356] "Generic (PLEG): container finished" podID="e92dd775-d49f-47b9-bf80-448ae55bb6f2" containerID="7b649a522aea98c35ebf35569890e580334eca9ac7d73ce447954cc45dd48828" exitCode=0 Jul 17 17:25:18.126919 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:18.126924 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjj97" event={"ID":"e92dd775-d49f-47b9-bf80-448ae55bb6f2","Type":"ContainerDied","Data":"7b649a522aea98c35ebf35569890e580334eca9ac7d73ce447954cc45dd48828"} Jul 17 17:25:18.258549 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:18.258518 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:18.357073 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:18.357043 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:18.411503 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:18.411443 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:18.555767 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:18.555735 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:18.771029 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:18.770994 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:18.771196 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:18.771033 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:18.771196 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:18.771125 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:25:18.771196 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:18.771159 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:25:18.771312 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:18.771202 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:18.771312 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:18.771273 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:25:18.937755 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:18.937724 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:18.998523 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:18.998476 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:19.131790 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:19.131708 2573 generic.go:356] "Generic (PLEG): container finished" podID="e92dd775-d49f-47b9-bf80-448ae55bb6f2" containerID="5bf53f28130582cba00b35d14b4dfae9b28e76a4f37fa09ec9ad422ca148573b" exitCode=0 Jul 17 17:25:19.131790 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:19.131756 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjj97" event={"ID":"e92dd775-d49f-47b9-bf80-448ae55bb6f2","Type":"ContainerDied","Data":"5bf53f28130582cba00b35d14b4dfae9b28e76a4f37fa09ec9ad422ca148573b"} Jul 17 17:25:19.339426 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:19.339399 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:19.441960 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:19.441876 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:19.561375 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:19.561345 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:19.619593 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:19.619567 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:19.949946 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:19.949904 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:20.137227 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:20.137197 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjj97" event={"ID":"e92dd775-d49f-47b9-bf80-448ae55bb6f2","Type":"ContainerStarted","Data":"9f77cc542746fd104102c65a84a595b82f5b1fa16673f7502ea2635e95d28a13"} Jul 17 17:25:20.149392 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:20.149366 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:20.163468 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:20.163425 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wjj97" podStartSLOduration=9.598564313 podStartE2EDuration="41.163412113s" totalImagesPullingTime="31.5648478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:24:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:20.161972948 +0000 UTC m=+40.897396997" watchObservedRunningTime="2026-07-17 17:25:20.163412113 +0000 UTC m=+40.898836162" Jul 17 17:25:20.637400 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:20.637364 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:20.692555 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:20.692520 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:20.771474 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:20.771440 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:20.771675 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:20.771541 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:20.771675 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:20.771548 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:25:20.771675 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:20.771547 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:20.771675 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:20.771659 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:25:20.771837 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:20.771694 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:25:20.961708 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:20.961591 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:21.059973 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:21.059945 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:22.561556 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:22.561520 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:22.690490 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:22.690443 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:22.771197 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:22.771163 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:22.771333 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:22.771203 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:22.771333 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:22.771181 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:22.771333 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:22.771279 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:25:22.771499 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:22.771332 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:25:22.771499 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:22.771410 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:25:22.789033 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:22.789009 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:22.890556 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:22.890492 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:22.988789 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:22.988762 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:23.867956 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:23.867767 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pvv46"] Jul 17 17:25:23.868311 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:23.868035 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:23.868311 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:23.868122 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:25:23.871034 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:23.871003 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w49zh"] Jul 17 17:25:23.871185 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:23.871114 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:23.871253 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:23.871207 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:25:23.871850 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:23.871828 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hr9d5"] Jul 17 17:25:23.871956 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:23.871907 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:23.872015 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:23.871995 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:25:24.360354 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:24.360325 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:24.426739 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:24.426712 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:24.965200 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:24.965169 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:25.020075 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:25.020046 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:25.771887 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:25.771858 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:25.771887 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:25.771889 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:25.772122 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:25.771860 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:25.772122 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:25.771985 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:25:25.772122 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:25.772107 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:25:25.772272 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:25.772221 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:25:27.771719 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:27.771683 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:27.771719 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:27.771712 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:27.772352 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:27.771683 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:27.772352 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:27.771799 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hr9d5" podUID="2a96d8c8-87eb-47c6-b427-0486a1b6eb23" Jul 17 17:25:27.772352 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:27.771859 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pvv46" podUID="b2859ee6-181c-4e9c-a3f6-777df52447fa" Jul 17 17:25:27.772352 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:27.771932 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w49zh" podUID="c0ca305d-1ff8-4cab-bde6-bc63609387aa" Jul 17 17:25:28.149548 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.149524 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-183.ec2.internal" event="NodeReady" Jul 17 17:25:28.149731 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.149656 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 17:25:28.193421 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.193393 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cf6c9"] Jul 17 17:25:28.212247 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.212223 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kpn4z"] Jul 17 17:25:28.212382 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.212343 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cf6c9" Jul 17 17:25:28.215045 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.215018 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 17:25:28.215186 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.215057 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-smb8x\"" Jul 17 17:25:28.215186 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.215057 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 17:25:28.229726 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.229703 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cf6c9"] Jul 17 17:25:28.229838 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.229732 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kpn4z"] Jul 17 17:25:28.229838 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.229831 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.235520 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.232951 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 17:25:28.235520 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.233115 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j69cs\"" Jul 17 17:25:28.235520 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.233609 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 17:25:28.235520 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.233720 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 17:25:28.235520 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.233828 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 17:25:28.295109 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.295079 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sstpk"] Jul 17 17:25:28.305767 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.305738 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e582980-4dc3-42ae-88f3-f9c09f0709c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpn4z\" (UID: \"3e582980-4dc3-42ae-88f3-f9c09f0709c4\") " pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.305767 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.305775 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e582980-4dc3-42ae-88f3-f9c09f0709c4-data-volume\") pod \"insights-runtime-extractor-kpn4z\" (UID: \"3e582980-4dc3-42ae-88f3-f9c09f0709c4\") " pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.306027 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.305802 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd76aa3-4339-43c6-a285-70b59ed5fd66-config-volume\") pod \"dns-default-cf6c9\" (UID: \"2fd76aa3-4339-43c6-a285-70b59ed5fd66\") " pod="openshift-dns/dns-default-cf6c9" Jul 17 17:25:28.306027 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.305834 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e582980-4dc3-42ae-88f3-f9c09f0709c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpn4z\" (UID: \"3e582980-4dc3-42ae-88f3-f9c09f0709c4\") " pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.306027 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.305926 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd76aa3-4339-43c6-a285-70b59ed5fd66-tmp-dir\") pod \"dns-default-cf6c9\" (UID: \"2fd76aa3-4339-43c6-a285-70b59ed5fd66\") " pod="openshift-dns/dns-default-cf6c9" Jul 17 17:25:28.306027 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.306007 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fd76aa3-4339-43c6-a285-70b59ed5fd66-metrics-tls\") pod \"dns-default-cf6c9\" (UID: \"2fd76aa3-4339-43c6-a285-70b59ed5fd66\") " pod="openshift-dns/dns-default-cf6c9" Jul 17 17:25:28.306204 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.306051 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89gmb\" (UniqueName: \"kubernetes.io/projected/2fd76aa3-4339-43c6-a285-70b59ed5fd66-kube-api-access-89gmb\") pod \"dns-default-cf6c9\" (UID: \"2fd76aa3-4339-43c6-a285-70b59ed5fd66\") " pod="openshift-dns/dns-default-cf6c9" Jul 17 17:25:28.306204 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.306100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e582980-4dc3-42ae-88f3-f9c09f0709c4-crio-socket\") pod \"insights-runtime-extractor-kpn4z\" (UID: \"3e582980-4dc3-42ae-88f3-f9c09f0709c4\") " pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.306204 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.306152 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rknkm\" (UniqueName: \"kubernetes.io/projected/3e582980-4dc3-42ae-88f3-f9c09f0709c4-kube-api-access-rknkm\") pod \"insights-runtime-extractor-kpn4z\" (UID: \"3e582980-4dc3-42ae-88f3-f9c09f0709c4\") " pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.310852 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.310830 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:28.315611 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.315595 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sstpk"] Jul 17 17:25:28.315734 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.315719 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sstpk" Jul 17 17:25:28.318365 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.318347 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mcwhs\"" Jul 17 17:25:28.318455 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.318385 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 17:25:28.318455 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.318353 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 17:25:28.318455 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.318427 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 17:25:28.406837 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.406765 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fd76aa3-4339-43c6-a285-70b59ed5fd66-metrics-tls\") pod \"dns-default-cf6c9\" (UID: \"2fd76aa3-4339-43c6-a285-70b59ed5fd66\") " pod="openshift-dns/dns-default-cf6c9" Jul 17 17:25:28.406837 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.406794 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89gmb\" (UniqueName: \"kubernetes.io/projected/2fd76aa3-4339-43c6-a285-70b59ed5fd66-kube-api-access-89gmb\") pod \"dns-default-cf6c9\" (UID: \"2fd76aa3-4339-43c6-a285-70b59ed5fd66\") " pod="openshift-dns/dns-default-cf6c9" Jul 17 17:25:28.406837 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.406826 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e582980-4dc3-42ae-88f3-f9c09f0709c4-crio-socket\") pod \"insights-runtime-extractor-kpn4z\" (UID: \"3e582980-4dc3-42ae-88f3-f9c09f0709c4\") " pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.407075 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.406874 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rknkm\" (UniqueName: \"kubernetes.io/projected/3e582980-4dc3-42ae-88f3-f9c09f0709c4-kube-api-access-rknkm\") pod \"insights-runtime-extractor-kpn4z\" (UID: \"3e582980-4dc3-42ae-88f3-f9c09f0709c4\") " pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.407075 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.406920 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e582980-4dc3-42ae-88f3-f9c09f0709c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpn4z\" (UID: \"3e582980-4dc3-42ae-88f3-f9c09f0709c4\") " pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.407075 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.406942 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66a066f3-5b00-49a6-8b43-7498b829ca1e-cert\") pod \"ingress-canary-sstpk\" (UID: \"66a066f3-5b00-49a6-8b43-7498b829ca1e\") " pod="openshift-ingress-canary/ingress-canary-sstpk" Jul 17 17:25:28.407075 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.407017 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d99rp\" (UniqueName: \"kubernetes.io/projected/66a066f3-5b00-49a6-8b43-7498b829ca1e-kube-api-access-d99rp\") pod \"ingress-canary-sstpk\" (UID: \"66a066f3-5b00-49a6-8b43-7498b829ca1e\") " pod="openshift-ingress-canary/ingress-canary-sstpk" Jul 17 17:25:28.407075 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.407062 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e582980-4dc3-42ae-88f3-f9c09f0709c4-data-volume\") pod \"insights-runtime-extractor-kpn4z\" (UID: \"3e582980-4dc3-42ae-88f3-f9c09f0709c4\") " pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.407322 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.407020 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e582980-4dc3-42ae-88f3-f9c09f0709c4-crio-socket\") pod \"insights-runtime-extractor-kpn4z\" (UID: \"3e582980-4dc3-42ae-88f3-f9c09f0709c4\") " pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.407322 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.407097 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd76aa3-4339-43c6-a285-70b59ed5fd66-config-volume\") pod \"dns-default-cf6c9\" (UID: \"2fd76aa3-4339-43c6-a285-70b59ed5fd66\") " pod="openshift-dns/dns-default-cf6c9" Jul 17 17:25:28.407322 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.407149 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e582980-4dc3-42ae-88f3-f9c09f0709c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpn4z\" (UID: \"3e582980-4dc3-42ae-88f3-f9c09f0709c4\") " pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.407322 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.407209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd76aa3-4339-43c6-a285-70b59ed5fd66-tmp-dir\") pod \"dns-default-cf6c9\" (UID: \"2fd76aa3-4339-43c6-a285-70b59ed5fd66\") " pod="openshift-dns/dns-default-cf6c9" Jul 17 17:25:28.407462 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.407424 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e582980-4dc3-42ae-88f3-f9c09f0709c4-data-volume\") pod \"insights-runtime-extractor-kpn4z\" (UID: \"3e582980-4dc3-42ae-88f3-f9c09f0709c4\") " pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.407502 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.407490 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd76aa3-4339-43c6-a285-70b59ed5fd66-tmp-dir\") pod \"dns-default-cf6c9\" (UID: \"2fd76aa3-4339-43c6-a285-70b59ed5fd66\") " pod="openshift-dns/dns-default-cf6c9" Jul 17 17:25:28.407535 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.407489 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e582980-4dc3-42ae-88f3-f9c09f0709c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpn4z\" (UID: \"3e582980-4dc3-42ae-88f3-f9c09f0709c4\") " pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.407732 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.407710 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd76aa3-4339-43c6-a285-70b59ed5fd66-config-volume\") pod \"dns-default-cf6c9\" (UID: \"2fd76aa3-4339-43c6-a285-70b59ed5fd66\") " pod="openshift-dns/dns-default-cf6c9" Jul 17 17:25:28.410873 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.410851 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fd76aa3-4339-43c6-a285-70b59ed5fd66-metrics-tls\") pod \"dns-default-cf6c9\" (UID: \"2fd76aa3-4339-43c6-a285-70b59ed5fd66\") " pod="openshift-dns/dns-default-cf6c9" Jul 17 17:25:28.410873 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.410867 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e582980-4dc3-42ae-88f3-f9c09f0709c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpn4z\" (UID: \"3e582980-4dc3-42ae-88f3-f9c09f0709c4\") " pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.411496 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.411478 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:28.414485 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.414458 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rknkm\" (UniqueName: \"kubernetes.io/projected/3e582980-4dc3-42ae-88f3-f9c09f0709c4-kube-api-access-rknkm\") pod \"insights-runtime-extractor-kpn4z\" (UID: \"3e582980-4dc3-42ae-88f3-f9c09f0709c4\") " pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.414660 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.414637 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89gmb\" (UniqueName: \"kubernetes.io/projected/2fd76aa3-4339-43c6-a285-70b59ed5fd66-kube-api-access-89gmb\") pod \"dns-default-cf6c9\" (UID: \"2fd76aa3-4339-43c6-a285-70b59ed5fd66\") " pod="openshift-dns/dns-default-cf6c9" Jul 17 17:25:28.507638 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.507598 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66a066f3-5b00-49a6-8b43-7498b829ca1e-cert\") pod \"ingress-canary-sstpk\" (UID: \"66a066f3-5b00-49a6-8b43-7498b829ca1e\") " pod="openshift-ingress-canary/ingress-canary-sstpk" Jul 17 17:25:28.507638 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.507644 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d99rp\" (UniqueName: \"kubernetes.io/projected/66a066f3-5b00-49a6-8b43-7498b829ca1e-kube-api-access-d99rp\") pod \"ingress-canary-sstpk\" (UID: \"66a066f3-5b00-49a6-8b43-7498b829ca1e\") " pod="openshift-ingress-canary/ingress-canary-sstpk" Jul 17 17:25:28.509999 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.509979 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66a066f3-5b00-49a6-8b43-7498b829ca1e-cert\") pod \"ingress-canary-sstpk\" (UID: \"66a066f3-5b00-49a6-8b43-7498b829ca1e\") " pod="openshift-ingress-canary/ingress-canary-sstpk" Jul 17 17:25:28.515591 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.515570 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d99rp\" (UniqueName: \"kubernetes.io/projected/66a066f3-5b00-49a6-8b43-7498b829ca1e-kube-api-access-d99rp\") pod \"ingress-canary-sstpk\" (UID: \"66a066f3-5b00-49a6-8b43-7498b829ca1e\") " pod="openshift-ingress-canary/ingress-canary-sstpk" Jul 17 17:25:28.521418 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.521399 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cf6c9" Jul 17 17:25:28.540550 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.540533 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kpn4z" Jul 17 17:25:28.624503 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.624468 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sstpk" Jul 17 17:25:28.722930 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.722880 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cf6c9"] Jul 17 17:25:28.726327 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.726295 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kpn4z"] Jul 17 17:25:28.727455 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:25:28.727422 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd76aa3_4339_43c6_a285_70b59ed5fd66.slice/crio-64705742efd8495b3d157103b73f3e305ffb4b49aa466dd585633398df8f5115 WatchSource:0}: Error finding container 64705742efd8495b3d157103b73f3e305ffb4b49aa466dd585633398df8f5115: Status 404 returned error can't find the container with id 64705742efd8495b3d157103b73f3e305ffb4b49aa466dd585633398df8f5115 Jul 17 17:25:28.729187 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:25:28.729161 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e582980_4dc3_42ae_88f3_f9c09f0709c4.slice/crio-3f77b7e744b3c198539a985d6f37ce30eaf07ca496faeb29f27563da7e457ea8 WatchSource:0}: Error finding container 3f77b7e744b3c198539a985d6f37ce30eaf07ca496faeb29f27563da7e457ea8: Status 404 returned error can't find the container with id 3f77b7e744b3c198539a985d6f37ce30eaf07ca496faeb29f27563da7e457ea8 Jul 17 17:25:28.781725 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:28.781536 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sstpk"] Jul 17 17:25:28.784215 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:25:28.784187 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a066f3_5b00_49a6_8b43_7498b829ca1e.slice/crio-56d5155f297c65d63ea088c1961d5da3f7b2477a9ab9b30f19d73c14b44f9327 WatchSource:0}: Error finding container 56d5155f297c65d63ea088c1961d5da3f7b2477a9ab9b30f19d73c14b44f9327: Status 404 returned error can't find the container with id 56d5155f297c65d63ea088c1961d5da3f7b2477a9ab9b30f19d73c14b44f9327 Jul 17 17:25:29.157064 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:29.157025 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sstpk" event={"ID":"66a066f3-5b00-49a6-8b43-7498b829ca1e","Type":"ContainerStarted","Data":"56d5155f297c65d63ea088c1961d5da3f7b2477a9ab9b30f19d73c14b44f9327"} Jul 17 17:25:29.158033 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:29.158005 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kpn4z" event={"ID":"3e582980-4dc3-42ae-88f3-f9c09f0709c4","Type":"ContainerStarted","Data":"3f77b7e744b3c198539a985d6f37ce30eaf07ca496faeb29f27563da7e457ea8"} Jul 17 17:25:29.158905 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:29.158884 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cf6c9" event={"ID":"2fd76aa3-4339-43c6-a285-70b59ed5fd66","Type":"ContainerStarted","Data":"64705742efd8495b3d157103b73f3e305ffb4b49aa466dd585633398df8f5115"} Jul 17 17:25:29.773601 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:29.773562 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:29.773796 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:29.773724 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:29.773796 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:29.773779 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:29.777903 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:29.777879 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 17:25:29.778042 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:29.777908 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-srf5x\"" Jul 17 17:25:29.778042 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:29.777929 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 17:25:29.778359 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:29.778063 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 17:25:29.778359 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:29.778119 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 17:25:29.778359 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:29.778171 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nlczc\"" Jul 17 17:25:30.165358 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:30.165287 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:30.265652 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:30.265523 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:31.587767 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.587729 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-94cdd4948-wj9zl"] Jul 17 17:25:31.603110 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.603043 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-94cdd4948-wj9zl"] Jul 17 17:25:31.603253 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.603153 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.606350 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.606318 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 17 17:25:31.606350 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.606329 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 17 17:25:31.606350 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.606341 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 17 17:25:31.606598 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.606408 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 17 17:25:31.607201 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.607179 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 17 17:25:31.607307 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.607228 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sx27b\"" Jul 17 17:25:31.607477 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.607458 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 17 17:25:31.607602 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.607481 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 17 17:25:31.630340 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.630316 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78c15898-e760-4e54-adc0-e13c7fa473a9-console-serving-cert\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.630462 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.630348 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxbhb\" (UniqueName: \"kubernetes.io/projected/78c15898-e760-4e54-adc0-e13c7fa473a9-kube-api-access-zxbhb\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.630462 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.630374 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-oauth-serving-cert\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.630576 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.630466 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78c15898-e760-4e54-adc0-e13c7fa473a9-console-oauth-config\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.630576 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.630495 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-console-config\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.630576 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.630522 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-service-ca\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.731119 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.731081 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78c15898-e760-4e54-adc0-e13c7fa473a9-console-serving-cert\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.731119 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.731125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbhb\" (UniqueName: \"kubernetes.io/projected/78c15898-e760-4e54-adc0-e13c7fa473a9-kube-api-access-zxbhb\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.731358 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.731148 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-oauth-serving-cert\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.731358 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.731186 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78c15898-e760-4e54-adc0-e13c7fa473a9-console-oauth-config\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.731358 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.731205 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-console-config\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.731358 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.731225 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-service-ca\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.732174 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.732142 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-console-config\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.732174 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.732151 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-oauth-serving-cert\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.732345 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.732151 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-service-ca\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.734574 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.734550 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78c15898-e760-4e54-adc0-e13c7fa473a9-console-oauth-config\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.734731 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.734715 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78c15898-e760-4e54-adc0-e13c7fa473a9-console-serving-cert\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.742358 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.742340 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxbhb\" (UniqueName: \"kubernetes.io/projected/78c15898-e760-4e54-adc0-e13c7fa473a9-kube-api-access-zxbhb\") pod \"console-94cdd4948-wj9zl\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:31.914578 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:31.914485 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:33.791143 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:33.791063 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:33.848193 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:33.848155 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:34.811436 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:34.811329 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-94cdd4948-wj9zl"] Jul 17 17:25:34.817185 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:25:34.816835 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c15898_e760_4e54_adc0_e13c7fa473a9.slice/crio-0a2752cf23b5f7645d58ee9f14f7887dc9336431f10a79e83e4e533e6fd0b72f WatchSource:0}: Error finding container 0a2752cf23b5f7645d58ee9f14f7887dc9336431f10a79e83e4e533e6fd0b72f: Status 404 returned error can't find the container with id 0a2752cf23b5f7645d58ee9f14f7887dc9336431f10a79e83e4e533e6fd0b72f Jul 17 17:25:35.175905 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:35.175865 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-94cdd4948-wj9zl" event={"ID":"78c15898-e760-4e54-adc0-e13c7fa473a9","Type":"ContainerStarted","Data":"0a2752cf23b5f7645d58ee9f14f7887dc9336431f10a79e83e4e533e6fd0b72f"} Jul 17 17:25:35.177401 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:35.177367 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sstpk" event={"ID":"66a066f3-5b00-49a6-8b43-7498b829ca1e","Type":"ContainerStarted","Data":"2dcbaed505963787fbe3733113e4e8bfbb1039f083d73cc247e567304e68640e"} Jul 17 17:25:35.178994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:35.178973 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kpn4z" event={"ID":"3e582980-4dc3-42ae-88f3-f9c09f0709c4","Type":"ContainerStarted","Data":"21ebe8e41aaf867216cb36f5ed407ab223ff4161bbb7af63fce5fb373a889f8b"} Jul 17 17:25:35.180849 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:35.180824 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cf6c9" event={"ID":"2fd76aa3-4339-43c6-a285-70b59ed5fd66","Type":"ContainerStarted","Data":"742d519f6a7f1b381bc18188e6544b0ee96b0c56845b620af07a6e231eb963e4"} Jul 17 17:25:35.180934 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:35.180868 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cf6c9" event={"ID":"2fd76aa3-4339-43c6-a285-70b59ed5fd66","Type":"ContainerStarted","Data":"ed53bb1b323c9f754d126e8323f2e228aed9615c811b4ae1debac8a9db5e8e2b"} Jul 17 17:25:35.181000 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:35.180981 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cf6c9" Jul 17 17:25:35.193340 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:35.193298 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sstpk" podStartSLOduration=1.311133185 podStartE2EDuration="7.193282022s" totalImagesPullingTime="5.882148837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:35.192885734 +0000 UTC m=+55.928309783" watchObservedRunningTime="2026-07-17 17:25:35.193282022 +0000 UTC m=+55.928706073" Jul 17 17:25:35.212739 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:35.212682 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cf6c9" podStartSLOduration=1.597333856 podStartE2EDuration="7.212662889s" totalImagesPullingTime="5.615329033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:35.211344119 +0000 UTC m=+55.946768169" watchObservedRunningTime="2026-07-17 17:25:35.212662889 +0000 UTC m=+55.948086937" Jul 17 17:25:35.938866 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:35.938827 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:38.575268 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.575234 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76f6c85f7-cs49h"] Jul 17 17:25:38.581747 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.581727 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.590008 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.589939 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76f6c85f7-cs49h"] Jul 17 17:25:38.591079 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.590693 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:38.591815 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.591795 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 17 17:25:38.643499 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.643452 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:38.682717 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.682673 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/458425f9-4fdd-453b-8b14-0fafcb83185f-console-oauth-config\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.682717 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.682719 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-console-config\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.682961 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.682787 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n496f\" (UniqueName: \"kubernetes.io/projected/458425f9-4fdd-453b-8b14-0fafcb83185f-kube-api-access-n496f\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.682961 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.682869 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-service-ca\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.682961 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.682901 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-trusted-ca-bundle\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.682961 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.682927 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-oauth-serving-cert\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.683109 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.683032 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/458425f9-4fdd-453b-8b14-0fafcb83185f-console-serving-cert\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.783842 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.783803 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/458425f9-4fdd-453b-8b14-0fafcb83185f-console-oauth-config\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.783842 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.783837 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-console-config\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.784073 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.783856 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n496f\" (UniqueName: \"kubernetes.io/projected/458425f9-4fdd-453b-8b14-0fafcb83185f-kube-api-access-n496f\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.784073 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.783872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-service-ca\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.784073 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.783886 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-trusted-ca-bundle\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.784073 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.783908 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-oauth-serving-cert\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.784260 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.784126 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/458425f9-4fdd-453b-8b14-0fafcb83185f-console-serving-cert\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.784799 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.784729 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-console-config\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.784799 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.784744 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-oauth-serving-cert\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.784958 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.784868 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-service-ca\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.784958 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.784944 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-trusted-ca-bundle\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.788599 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.788573 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/458425f9-4fdd-453b-8b14-0fafcb83185f-console-oauth-config\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.788730 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.788578 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/458425f9-4fdd-453b-8b14-0fafcb83185f-console-serving-cert\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.791604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.791584 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n496f\" (UniqueName: \"kubernetes.io/projected/458425f9-4fdd-453b-8b14-0fafcb83185f-kube-api-access-n496f\") pod \"console-76f6c85f7-cs49h\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:38.894055 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:38.893972 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:39.108872 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:39.108841 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7lgg2" Jul 17 17:25:39.234563 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:39.234529 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:39.307634 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:39.307588 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:39.453423 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:39.453395 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76f6c85f7-cs49h"] Jul 17 17:25:39.456559 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:25:39.456536 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458425f9_4fdd_453b_8b14_0fafcb83185f.slice/crio-0fb9e04c3a55e20ac9b278f2aa4c75772908aa2f0a241741144c45f2eb2b8e9c WatchSource:0}: Error finding container 0fb9e04c3a55e20ac9b278f2aa4c75772908aa2f0a241741144c45f2eb2b8e9c: Status 404 returned error can't find the container with id 0fb9e04c3a55e20ac9b278f2aa4c75772908aa2f0a241741144c45f2eb2b8e9c Jul 17 17:25:40.198261 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:40.198223 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kpn4z" event={"ID":"3e582980-4dc3-42ae-88f3-f9c09f0709c4","Type":"ContainerStarted","Data":"a6d07ab60c1943de0f199113241eaaa3b09039758eaaa033425ed9b4113f054f"} Jul 17 17:25:40.199964 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:40.199933 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f6c85f7-cs49h" event={"ID":"458425f9-4fdd-453b-8b14-0fafcb83185f","Type":"ContainerStarted","Data":"10fa84f945d288d8771f216a00db87e7b02e3db6bae6e89546bade05a8161641"} Jul 17 17:25:40.200088 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:40.199983 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f6c85f7-cs49h" event={"ID":"458425f9-4fdd-453b-8b14-0fafcb83185f","Type":"ContainerStarted","Data":"0fb9e04c3a55e20ac9b278f2aa4c75772908aa2f0a241741144c45f2eb2b8e9c"} Jul 17 17:25:40.201507 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:40.201481 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-94cdd4948-wj9zl" event={"ID":"78c15898-e760-4e54-adc0-e13c7fa473a9","Type":"ContainerStarted","Data":"7f7ff0e39230fc737ee98fc4843026a62253757c278e71a74efb90a27f4dbe22"} Jul 17 17:25:40.217224 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:40.217173 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76f6c85f7-cs49h" podStartSLOduration=2.217154839 podStartE2EDuration="2.217154839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:40.216516956 +0000 UTC m=+60.951941007" watchObservedRunningTime="2026-07-17 17:25:40.217154839 +0000 UTC m=+60.952578889" Jul 17 17:25:40.234685 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:40.234632 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-94cdd4948-wj9zl" podStartSLOduration=4.726978675 podStartE2EDuration="9.234599882s" totalImagesPullingTime="4.507621207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:25:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:40.233373437 +0000 UTC m=+60.968797489" watchObservedRunningTime="2026-07-17 17:25:40.234599882 +0000 UTC m=+60.970023932" Jul 17 17:25:41.002198 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:41.002163 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:41.066237 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:41.066199 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:41.915650 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:41.915599 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:41.916104 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:41.915691 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:41.921335 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:41.921309 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:42.213352 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:42.213277 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:25:42.235506 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:42.235466 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:42.334173 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:42.334118 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:43.214152 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:43.214118 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kpn4z" event={"ID":"3e582980-4dc3-42ae-88f3-f9c09f0709c4","Type":"ContainerStarted","Data":"7d82611af07bb7c6d5e274614e4751d32c2ce862d89276bdb756568bb83dd763"} Jul 17 17:25:43.231125 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:43.231081 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kpn4z" podStartSLOduration=1.071485273 podStartE2EDuration="15.23106546s" totalImagesPullingTime="14.159580187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:25:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:43.23059899 +0000 UTC m=+63.966023040" watchObservedRunningTime="2026-07-17 17:25:43.23106546 +0000 UTC m=+63.966489508" Jul 17 17:25:44.249824 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.249791 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:44.312569 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.312541 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:44.432233 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.432196 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs\") pod \"network-metrics-daemon-w49zh\" (UID: \"c0ca305d-1ff8-4cab-bde6-bc63609387aa\") " pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:44.435184 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.435166 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 17:25:44.445104 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.445081 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ca305d-1ff8-4cab-bde6-bc63609387aa-metrics-certs\") pod \"network-metrics-daemon-w49zh\" (UID: \"c0ca305d-1ff8-4cab-bde6-bc63609387aa\") " pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:44.450566 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.450545 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:44.490948 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.490932 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nlczc\"" Jul 17 17:25:44.497490 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.497473 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w49zh" Jul 17 17:25:44.507615 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.507595 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:44.532895 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.532864 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret\") pod \"global-pull-secret-syncer-hr9d5\" (UID: \"2a96d8c8-87eb-47c6-b427-0486a1b6eb23\") " pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:44.533047 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.532914 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77pt\" (UniqueName: \"kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt\") pod \"network-check-target-pvv46\" (UID: \"b2859ee6-181c-4e9c-a3f6-777df52447fa\") " pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:44.535959 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.535938 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 17:25:44.536069 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.535965 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 17:25:44.545911 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.545879 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 17:25:44.546208 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.546102 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a96d8c8-87eb-47c6-b427-0486a1b6eb23-original-pull-secret\") pod \"global-pull-secret-syncer-hr9d5\" (UID: \"2a96d8c8-87eb-47c6-b427-0486a1b6eb23\") " pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:44.556987 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.556962 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v77pt\" (UniqueName: \"kubernetes.io/projected/b2859ee6-181c-4e9c-a3f6-777df52447fa-kube-api-access-v77pt\") pod \"network-check-target-pvv46\" (UID: \"b2859ee6-181c-4e9c-a3f6-777df52447fa\") " pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:44.616596 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.616568 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w49zh"] Jul 17 17:25:44.619790 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:25:44.619766 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ca305d_1ff8_4cab_bde6_bc63609387aa.slice/crio-4139e5753dc3dd7c5d97c8ead448b7dd5124460e809dee27b4adade127d942e8 WatchSource:0}: Error finding container 4139e5753dc3dd7c5d97c8ead448b7dd5124460e809dee27b4adade127d942e8: Status 404 returned error can't find the container with id 4139e5753dc3dd7c5d97c8ead448b7dd5124460e809dee27b4adade127d942e8 Jul 17 17:25:44.796261 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.796155 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hr9d5" Jul 17 17:25:44.806650 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.806608 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-srf5x\"" Jul 17 17:25:44.814912 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.814887 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:44.924988 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.924960 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hr9d5"] Jul 17 17:25:44.929002 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:25:44.928979 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a96d8c8_87eb_47c6_b427_0486a1b6eb23.slice/crio-894a23cb5a1bfc2325c714f25def37bfc9fa87f650f170a8a02477ceef7b64bd WatchSource:0}: Error finding container 894a23cb5a1bfc2325c714f25def37bfc9fa87f650f170a8a02477ceef7b64bd: Status 404 returned error can't find the container with id 894a23cb5a1bfc2325c714f25def37bfc9fa87f650f170a8a02477ceef7b64bd Jul 17 17:25:44.943440 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:44.943416 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pvv46"] Jul 17 17:25:44.946027 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:25:44.946007 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2859ee6_181c_4e9c_a3f6_777df52447fa.slice/crio-0826029d59553a204b34245fd2170d232f2b8a7928efeddf772902b33bdfb7f0 WatchSource:0}: Error finding container 0826029d59553a204b34245fd2170d232f2b8a7928efeddf772902b33bdfb7f0: Status 404 returned error can't find the container with id 0826029d59553a204b34245fd2170d232f2b8a7928efeddf772902b33bdfb7f0 Jul 17 17:25:45.186458 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.186377 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cf6c9" Jul 17 17:25:45.222135 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.222099 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hr9d5" event={"ID":"2a96d8c8-87eb-47c6-b427-0486a1b6eb23","Type":"ContainerStarted","Data":"894a23cb5a1bfc2325c714f25def37bfc9fa87f650f170a8a02477ceef7b64bd"} Jul 17 17:25:45.223422 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.223391 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w49zh" event={"ID":"c0ca305d-1ff8-4cab-bde6-bc63609387aa","Type":"ContainerStarted","Data":"4139e5753dc3dd7c5d97c8ead448b7dd5124460e809dee27b4adade127d942e8"} Jul 17 17:25:45.224139 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.224108 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:45.224672 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.224639 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pvv46" event={"ID":"b2859ee6-181c-4e9c-a3f6-777df52447fa","Type":"ContainerStarted","Data":"0826029d59553a204b34245fd2170d232f2b8a7928efeddf772902b33bdfb7f0"} Jul 17 17:25:45.330685 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.330650 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:45.425936 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.425906 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:45.559269 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.559237 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:45.567179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.567145 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lnkmx"] Jul 17 17:25:45.569449 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.569420 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lnkmx" Jul 17 17:25:45.571985 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.571964 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 17 17:25:45.571985 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.571981 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jkqrn\"" Jul 17 17:25:45.580471 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.580178 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lnkmx"] Jul 17 17:25:45.641946 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.641830 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c5a3ccba-9a63-4a3d-b7c5-9713e00881f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-lnkmx\" (UID: \"c5a3ccba-9a63-4a3d-b7c5-9713e00881f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lnkmx" Jul 17 17:25:45.656863 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.656838 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:45.743125 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.743007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c5a3ccba-9a63-4a3d-b7c5-9713e00881f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-lnkmx\" (UID: \"c5a3ccba-9a63-4a3d-b7c5-9713e00881f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lnkmx" Jul 17 17:25:45.743278 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:45.743188 2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 17 17:25:45.743278 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:45.743251 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a3ccba-9a63-4a3d-b7c5-9713e00881f5-tls-certificates podName:c5a3ccba-9a63-4a3d-b7c5-9713e00881f5 nodeName:}" failed. No retries permitted until 2026-07-17 17:25:46.24323185 +0000 UTC m=+66.978655878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/c5a3ccba-9a63-4a3d-b7c5-9713e00881f5-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-lnkmx" (UID: "c5a3ccba-9a63-4a3d-b7c5-9713e00881f5") : secret "prometheus-operator-admission-webhook-tls" not found Jul 17 17:25:45.756472 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:45.756442 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:46.114183 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:46.114107 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:46.235018 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:46.234948 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w49zh" event={"ID":"c0ca305d-1ff8-4cab-bde6-bc63609387aa","Type":"ContainerStarted","Data":"764cd3bcfcd48a058156e3b271d4db20ec67ba9bdac53c09216127bb8fcb5c4f"} Jul 17 17:25:46.235221 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:46.235049 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w49zh" event={"ID":"c0ca305d-1ff8-4cab-bde6-bc63609387aa","Type":"ContainerStarted","Data":"52173a1fc53bf916378f444af0956c8a413bc79e3460b34fdcadf9ee0d049a37"} Jul 17 17:25:46.246766 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:46.246733 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c5a3ccba-9a63-4a3d-b7c5-9713e00881f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-lnkmx\" (UID: \"c5a3ccba-9a63-4a3d-b7c5-9713e00881f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lnkmx" Jul 17 17:25:46.249654 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:46.249605 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c5a3ccba-9a63-4a3d-b7c5-9713e00881f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-lnkmx\" (UID: \"c5a3ccba-9a63-4a3d-b7c5-9713e00881f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lnkmx" Jul 17 17:25:46.253167 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:46.253119 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-w49zh" podStartSLOduration=65.385248013 podStartE2EDuration="1m6.253100926s" totalImagesPullingTime="867.852913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:46.252560455 +0000 UTC m=+66.987984525" watchObservedRunningTime="2026-07-17 17:25:46.253100926 +0000 UTC m=+66.988524967" Jul 17 17:25:46.348383 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:46.348352 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:46.426693 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:46.426573 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:46.481818 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:46.481668 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lnkmx" Jul 17 17:25:46.633442 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:46.633411 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lnkmx"] Jul 17 17:25:46.637325 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:25:46.637292 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a3ccba_9a63_4a3d_b7c5_9713e00881f5.slice/crio-44df668da74e279e3226d33b498331fb89026f7db9b7e8ea9a10dc11cb4b667b WatchSource:0}: Error finding container 44df668da74e279e3226d33b498331fb89026f7db9b7e8ea9a10dc11cb4b667b: Status 404 returned error can't find the container with id 44df668da74e279e3226d33b498331fb89026f7db9b7e8ea9a10dc11cb4b667b Jul 17 17:25:47.241100 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:47.241032 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lnkmx" event={"ID":"c5a3ccba-9a63-4a3d-b7c5-9713e00881f5","Type":"ContainerStarted","Data":"44df668da74e279e3226d33b498331fb89026f7db9b7e8ea9a10dc11cb4b667b"} Jul 17 17:25:48.894154 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:48.894114 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:48.894607 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:48.894189 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:48.899556 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:48.899529 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:49.253743 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:49.253711 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:25:49.299685 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:49.299656 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-94cdd4948-wj9zl"] Jul 17 17:25:49.408486 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:49.408450 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:49.459767 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:49.459734 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:49.525804 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:49.525727 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:50.253752 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:50.253720 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pvv46" event={"ID":"b2859ee6-181c-4e9c-a3f6-777df52447fa","Type":"ContainerStarted","Data":"f498ce4ae0a81040027864c725bddd9f6dc474dadd379aa088c74bd6e83e4c21"} Jul 17 17:25:50.254199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:50.253834 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:25:50.255177 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:50.255156 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hr9d5" event={"ID":"2a96d8c8-87eb-47c6-b427-0486a1b6eb23","Type":"ContainerStarted","Data":"2b65b32f8d8f7e69df3d25ee30d23dc2468520e0943a9e903571a97ee562d33e"} Jul 17 17:25:50.256579 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:50.256555 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lnkmx" event={"ID":"c5a3ccba-9a63-4a3d-b7c5-9713e00881f5","Type":"ContainerStarted","Data":"0556d7ca497080d46017e62fd4eb1d581bab96fcd8e1314d0e2dd9414a4ce1a6"} Jul 17 17:25:50.269852 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:50.269807 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pvv46" podStartSLOduration=66.16467832 podStartE2EDuration="1m11.269791687s" totalImagesPullingTime="5.105113367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:50.269185172 +0000 UTC m=+71.004609220" watchObservedRunningTime="2026-07-17 17:25:50.269791687 +0000 UTC m=+71.005215735" Jul 17 17:25:50.284202 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:50.284158 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hr9d5" podStartSLOduration=65.147685585 podStartE2EDuration="1m10.284145156s" totalImagesPullingTime="5.136459571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:50.283223112 +0000 UTC m=+71.018647161" watchObservedRunningTime="2026-07-17 17:25:50.284145156 +0000 UTC m=+71.019569205" Jul 17 17:25:50.298668 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:50.298556 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lnkmx" podStartSLOduration=1.886181849 podStartE2EDuration="5.298541837s" totalImagesPullingTime="3.412359988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:50.297748117 +0000 UTC m=+71.033172167" watchObservedRunningTime="2026-07-17 17:25:50.298541837 +0000 UTC m=+71.033965887" Jul 17 17:25:51.259849 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.259812 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lnkmx" Jul 17 17:25:51.264277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.264257 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lnkmx" Jul 17 17:25:51.589855 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.589772 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:51.629762 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.629737 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl"] Jul 17 17:25:51.631587 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.631573 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" Jul 17 17:25:51.633952 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.633934 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 17:25:51.635143 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.635124 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 17 17:25:51.635215 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.635152 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 17:25:51.635215 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.635163 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 17 17:25:51.635215 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.635162 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-b5hcd\"" Jul 17 17:25:51.635215 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.635128 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 17:25:51.640732 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.640714 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl"] Jul 17 17:25:51.660146 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.660122 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:51.687836 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.687815 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-bbkrl\" (UID: \"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" Jul 17 17:25:51.687915 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.687860 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-bbkrl\" (UID: \"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" Jul 17 17:25:51.687915 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.687882 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lmzh\" (UniqueName: \"kubernetes.io/projected/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-kube-api-access-2lmzh\") pod \"prometheus-operator-5945b5cfd-bbkrl\" (UID: \"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" Jul 17 17:25:51.687996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.687957 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-bbkrl\" (UID: \"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" Jul 17 17:25:51.788479 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.788448 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-bbkrl\" (UID: \"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" Jul 17 17:25:51.788650 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.788503 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-bbkrl\" (UID: \"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" Jul 17 17:25:51.788650 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.788532 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lmzh\" (UniqueName: \"kubernetes.io/projected/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-kube-api-access-2lmzh\") pod \"prometheus-operator-5945b5cfd-bbkrl\" (UID: \"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" Jul 17 17:25:51.788650 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.788552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-bbkrl\" (UID: \"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" Jul 17 17:25:51.788828 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:51.788707 2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 17 17:25:51.788828 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:51.788781 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-prometheus-operator-tls podName:d2e961a8-8a7c-44fa-b5db-86fc6fd95bec nodeName:}" failed. No retries permitted until 2026-07-17 17:25:52.288760099 +0000 UTC m=+73.024184144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-bbkrl" (UID: "d2e961a8-8a7c-44fa-b5db-86fc6fd95bec") : secret "prometheus-operator-tls" not found Jul 17 17:25:51.789121 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.789102 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-bbkrl\" (UID: \"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" Jul 17 17:25:51.791038 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.791016 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-bbkrl\" (UID: \"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" Jul 17 17:25:51.797132 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:51.797111 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lmzh\" (UniqueName: \"kubernetes.io/projected/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-kube-api-access-2lmzh\") pod \"prometheus-operator-5945b5cfd-bbkrl\" (UID: \"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" Jul 17 17:25:52.291078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:52.291032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-bbkrl\" (UID: \"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" Jul 17 17:25:52.291454 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:52.291178 2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 17 17:25:52.291454 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:52.291237 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-prometheus-operator-tls podName:d2e961a8-8a7c-44fa-b5db-86fc6fd95bec nodeName:}" failed. No retries permitted until 2026-07-17 17:25:53.291222607 +0000 UTC m=+74.026646634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-bbkrl" (UID: "d2e961a8-8a7c-44fa-b5db-86fc6fd95bec") : secret "prometheus-operator-tls" not found Jul 17 17:25:52.525673 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:52.525641 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:53.298045 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:53.298013 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-bbkrl\" (UID: \"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" Jul 17 17:25:53.300343 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:53.300319 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2e961a8-8a7c-44fa-b5db-86fc6fd95bec-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-bbkrl\" (UID: \"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" Jul 17 17:25:53.441093 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:53.441044 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" Jul 17 17:25:53.583750 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:53.583727 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl"] Jul 17 17:25:53.585881 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:25:53.585854 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e961a8_8a7c_44fa_b5db_86fc6fd95bec.slice/crio-6160008942f07f955f2d20bc0a698e37bc626ceae8183fa452e6030e9e525e9e WatchSource:0}: Error finding container 6160008942f07f955f2d20bc0a698e37bc626ceae8183fa452e6030e9e525e9e: Status 404 returned error can't find the container with id 6160008942f07f955f2d20bc0a698e37bc626ceae8183fa452e6030e9e525e9e Jul 17 17:25:54.272349 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:54.272320 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" event={"ID":"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec","Type":"ContainerStarted","Data":"6160008942f07f955f2d20bc0a698e37bc626ceae8183fa452e6030e9e525e9e"} Jul 17 17:25:54.357718 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:54.357680 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:54.437442 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:54.437407 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:54.987579 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:54.987552 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:55.277834 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:55.277741 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" event={"ID":"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec","Type":"ContainerStarted","Data":"26a76db9c2a34d99f684bdd0c7b8f06764e8ca30ba50afa52f5ec8815ce5e04e"} Jul 17 17:25:55.277834 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:55.277796 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" event={"ID":"d2e961a8-8a7c-44fa-b5db-86fc6fd95bec","Type":"ContainerStarted","Data":"d2343dfc06ef256796279d1f06b6f9db7cb097b019bd94eb9952e92ba27c15d6"} Jul 17 17:25:55.298705 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:55.298666 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bbkrl" podStartSLOduration=3.178774917 podStartE2EDuration="4.298652222s" totalImagesPullingTime="1.119877305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:55.297485319 +0000 UTC m=+76.032909389" watchObservedRunningTime="2026-07-17 17:25:55.298652222 +0000 UTC m=+76.034076271" Jul 17 17:25:55.311539 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:55.311506 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:55.412454 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:55.412430 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:56.739100 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:56.739064 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:56.838081 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:56.838050 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:57.015269 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.015205 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:57.015269 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.015237 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx"] Jul 17 17:25:57.017472 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.017458 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" Jul 17 17:25:57.020557 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.020537 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 17 17:25:57.020699 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.020615 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 17 17:25:57.020699 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.020640 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-drsmt\"" Jul 17 17:25:57.062000 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.061976 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx"] Jul 17 17:25:57.098228 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.098194 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-tb5v5"] Jul 17 17:25:57.102181 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.101782 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.109135 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.109071 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 17 17:25:57.109424 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.109304 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 17 17:25:57.109424 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.109345 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 17 17:25:57.109758 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.109669 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-d9zlf\"" Jul 17 17:25:57.110169 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.110149 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gt246"] Jul 17 17:25:57.112194 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.112174 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.126272 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.126253 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pgf2\" (UniqueName: \"kubernetes.io/projected/04577b57-1d9c-49f0-bee5-55cbb5cf8d9b-kube-api-access-7pgf2\") pod \"openshift-state-metrics-79f6958cdf-wnqgx\" (UID: \"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" Jul 17 17:25:57.126376 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.126281 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/04577b57-1d9c-49f0-bee5-55cbb5cf8d9b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-wnqgx\" (UID: \"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" Jul 17 17:25:57.126376 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.126313 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04577b57-1d9c-49f0-bee5-55cbb5cf8d9b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-wnqgx\" (UID: \"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" Jul 17 17:25:57.126486 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.126402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04577b57-1d9c-49f0-bee5-55cbb5cf8d9b-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-wnqgx\" (UID: \"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" Jul 17 17:25:57.131241 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.131223 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 17:25:57.132237 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.132215 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 17:25:57.132406 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.132390 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 17:25:57.132557 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.132541 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2zmsl\"" Jul 17 17:25:57.145533 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.145456 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-tb5v5"] Jul 17 17:25:57.227096 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227071 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dn4t\" (UniqueName: \"kubernetes.io/projected/8a63170d-0b4c-466d-8171-8e08637042d5-kube-api-access-5dn4t\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.227096 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227102 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a63170d-0b4c-466d-8171-8e08637042d5-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.227278 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227124 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a63170d-0b4c-466d-8171-8e08637042d5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.227278 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227150 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6a2178f-b9d1-48d5-963d-49d79be84b3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.227278 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227207 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04577b57-1d9c-49f0-bee5-55cbb5cf8d9b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-wnqgx\" (UID: \"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" Jul 17 17:25:57.227278 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227240 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8a63170d-0b4c-466d-8171-8e08637042d5-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.227278 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227275 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6a2178f-b9d1-48d5-963d-49d79be84b3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.227431 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227318 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6a2178f-b9d1-48d5-963d-49d79be84b3b-node-exporter-textfile\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.227431 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227334 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6a2178f-b9d1-48d5-963d-49d79be84b3b-node-exporter-wtmp\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.227431 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227355 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04577b57-1d9c-49f0-bee5-55cbb5cf8d9b-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-wnqgx\" (UID: \"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" Jul 17 17:25:57.227431 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227394 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a63170d-0b4c-466d-8171-8e08637042d5-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.227633 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227477 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ldsg\" (UniqueName: \"kubernetes.io/projected/f6a2178f-b9d1-48d5-963d-49d79be84b3b-kube-api-access-8ldsg\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.227633 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227509 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgf2\" (UniqueName: \"kubernetes.io/projected/04577b57-1d9c-49f0-bee5-55cbb5cf8d9b-kube-api-access-7pgf2\") pod \"openshift-state-metrics-79f6958cdf-wnqgx\" (UID: \"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" Jul 17 17:25:57.227633 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227525 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6a2178f-b9d1-48d5-963d-49d79be84b3b-node-exporter-tls\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.227633 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227551 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8a63170d-0b4c-466d-8171-8e08637042d5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.227633 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227571 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6a2178f-b9d1-48d5-963d-49d79be84b3b-sys\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.227873 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227653 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/04577b57-1d9c-49f0-bee5-55cbb5cf8d9b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-wnqgx\" (UID: \"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" Jul 17 17:25:57.227873 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227688 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6a2178f-b9d1-48d5-963d-49d79be84b3b-root\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.227873 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.227718 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6a2178f-b9d1-48d5-963d-49d79be84b3b-metrics-client-ca\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.227873 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:57.227794 2573 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 17 17:25:57.227873 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:25:57.227849 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04577b57-1d9c-49f0-bee5-55cbb5cf8d9b-openshift-state-metrics-tls podName:04577b57-1d9c-49f0-bee5-55cbb5cf8d9b nodeName:}" failed. No retries permitted until 2026-07-17 17:25:57.727835202 +0000 UTC m=+78.463259234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/04577b57-1d9c-49f0-bee5-55cbb5cf8d9b-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-wnqgx" (UID: "04577b57-1d9c-49f0-bee5-55cbb5cf8d9b") : secret "openshift-state-metrics-tls" not found Jul 17 17:25:57.228033 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.228009 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04577b57-1d9c-49f0-bee5-55cbb5cf8d9b-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-wnqgx\" (UID: \"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" Jul 17 17:25:57.229651 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.229631 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04577b57-1d9c-49f0-bee5-55cbb5cf8d9b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-wnqgx\" (UID: \"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" Jul 17 17:25:57.234544 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.234522 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:57.236927 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.236901 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pgf2\" (UniqueName: \"kubernetes.io/projected/04577b57-1d9c-49f0-bee5-55cbb5cf8d9b-kube-api-access-7pgf2\") pod \"openshift-state-metrics-79f6958cdf-wnqgx\" (UID: \"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" Jul 17 17:25:57.324817 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.324736 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:57.328861 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.328840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6a2178f-b9d1-48d5-963d-49d79be84b3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.328950 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.328872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6a2178f-b9d1-48d5-963d-49d79be84b3b-node-exporter-textfile\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.328950 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.328888 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6a2178f-b9d1-48d5-963d-49d79be84b3b-node-exporter-wtmp\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.328950 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.328906 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a63170d-0b4c-466d-8171-8e08637042d5-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.328950 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.328947 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ldsg\" (UniqueName: \"kubernetes.io/projected/f6a2178f-b9d1-48d5-963d-49d79be84b3b-kube-api-access-8ldsg\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.329145 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.328977 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6a2178f-b9d1-48d5-963d-49d79be84b3b-node-exporter-tls\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.329145 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.328998 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8a63170d-0b4c-466d-8171-8e08637042d5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.329145 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.329021 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6a2178f-b9d1-48d5-963d-49d79be84b3b-sys\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.329145 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.329061 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6a2178f-b9d1-48d5-963d-49d79be84b3b-node-exporter-wtmp\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.329145 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.329082 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6a2178f-b9d1-48d5-963d-49d79be84b3b-sys\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.329415 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.329163 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6a2178f-b9d1-48d5-963d-49d79be84b3b-root\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.329415 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.329192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6a2178f-b9d1-48d5-963d-49d79be84b3b-metrics-client-ca\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.329415 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.329241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dn4t\" (UniqueName: \"kubernetes.io/projected/8a63170d-0b4c-466d-8171-8e08637042d5-kube-api-access-5dn4t\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.329415 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.329268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a63170d-0b4c-466d-8171-8e08637042d5-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.329415 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.329328 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6a2178f-b9d1-48d5-963d-49d79be84b3b-node-exporter-textfile\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.329761 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.329615 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6a2178f-b9d1-48d5-963d-49d79be84b3b-root\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.329761 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.329666 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6a2178f-b9d1-48d5-963d-49d79be84b3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.329761 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.329673 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a63170d-0b4c-466d-8171-8e08637042d5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.329761 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.329728 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6a2178f-b9d1-48d5-963d-49d79be84b3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.329959 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.329771 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8a63170d-0b4c-466d-8171-8e08637042d5-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.329959 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.329797 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6a2178f-b9d1-48d5-963d-49d79be84b3b-metrics-client-ca\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.329959 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.329849 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8a63170d-0b4c-466d-8171-8e08637042d5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.329959 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.329885 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a63170d-0b4c-466d-8171-8e08637042d5-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.330162 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.330029 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8a63170d-0b4c-466d-8171-8e08637042d5-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.331545 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.331525 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6a2178f-b9d1-48d5-963d-49d79be84b3b-node-exporter-tls\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.331673 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.331594 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a63170d-0b4c-466d-8171-8e08637042d5-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.331919 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.331898 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6a2178f-b9d1-48d5-963d-49d79be84b3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.331965 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.331952 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a63170d-0b4c-466d-8171-8e08637042d5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.338958 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.338937 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dn4t\" (UniqueName: \"kubernetes.io/projected/8a63170d-0b4c-466d-8171-8e08637042d5-kube-api-access-5dn4t\") pod \"kube-state-metrics-56c5594674-tb5v5\" (UID: \"8a63170d-0b4c-466d-8171-8e08637042d5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.339079 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.339061 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ldsg\" (UniqueName: \"kubernetes.io/projected/f6a2178f-b9d1-48d5-963d-49d79be84b3b-kube-api-access-8ldsg\") pod \"node-exporter-gt246\" (UID: \"f6a2178f-b9d1-48d5-963d-49d79be84b3b\") " pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.389599 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.389556 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:57.412746 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.412723 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" Jul 17 17:25:57.419693 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.419678 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gt246" Jul 17 17:25:57.429824 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:25:57.429798 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a2178f_b9d1_48d5_963d_49d79be84b3b.slice/crio-c0c9b8ddc9a527643fd63821e21f9e701c130def2f896f9fde6ea0e51a05da85 WatchSource:0}: Error finding container c0c9b8ddc9a527643fd63821e21f9e701c130def2f896f9fde6ea0e51a05da85: Status 404 returned error can't find the container with id c0c9b8ddc9a527643fd63821e21f9e701c130def2f896f9fde6ea0e51a05da85 Jul 17 17:25:57.490840 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.490807 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:57.538763 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.538657 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-tb5v5"] Jul 17 17:25:57.541194 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:25:57.541164 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a63170d_0b4c_466d_8171_8e08637042d5.slice/crio-9e4c91e3bae0da700b65d0b52be6546921e67f1a9c27528add2fe6aeaa39b0ae WatchSource:0}: Error finding container 9e4c91e3bae0da700b65d0b52be6546921e67f1a9c27528add2fe6aeaa39b0ae: Status 404 returned error can't find the container with id 9e4c91e3bae0da700b65d0b52be6546921e67f1a9c27528add2fe6aeaa39b0ae Jul 17 17:25:57.607958 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.607879 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:57.732695 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.732658 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/04577b57-1d9c-49f0-bee5-55cbb5cf8d9b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-wnqgx\" (UID: \"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" Jul 17 17:25:57.734968 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.734952 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/04577b57-1d9c-49f0-bee5-55cbb5cf8d9b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-wnqgx\" (UID: \"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" Jul 17 17:25:57.925696 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.925601 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" Jul 17 17:25:57.995252 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:57.995223 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:58.085596 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.085562 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx"] Jul 17 17:25:58.088563 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:25:58.088530 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04577b57_1d9c_49f0_bee5_55cbb5cf8d9b.slice/crio-0defa7c30fcafc5990fd657477a32975f6e8985377c9dec4f63e348a3cd533f0 WatchSource:0}: Error finding container 0defa7c30fcafc5990fd657477a32975f6e8985377c9dec4f63e348a3cd533f0: Status 404 returned error can't find the container with id 0defa7c30fcafc5990fd657477a32975f6e8985377c9dec4f63e348a3cd533f0 Jul 17 17:25:58.088742 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.088713 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:58.101483 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.101456 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:25:58.105354 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.105334 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.109713 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.107762 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 17 17:25:58.109713 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.108130 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 17 17:25:58.109713 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.108420 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 17 17:25:58.109713 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.108948 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 17 17:25:58.109713 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.109236 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 17 17:25:58.109713 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.109445 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 17 17:25:58.109713 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.109612 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 17 17:25:58.109713 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.109663 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 17 17:25:58.110121 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.109999 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9fjng\"" Jul 17 17:25:58.110207 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.110190 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 17 17:25:58.124637 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.124594 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:25:58.237249 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.237214 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.237356 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.237264 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-config-volume\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.237356 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.237292 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zggwt\" (UniqueName: \"kubernetes.io/projected/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-kube-api-access-zggwt\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.237356 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.237322 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-config-out\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.237356 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.237346 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.237572 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.237372 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.237572 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.237428 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-web-config\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.237572 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.237475 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.237572 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.237511 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.237572 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.237533 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.237572 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.237570 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.237841 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.237599 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.237841 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.237682 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.290502 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.290466 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" event={"ID":"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b","Type":"ContainerStarted","Data":"f34353ccf857407d7dc69bbffa832a7c8b8366bf963c28c68b3da2b839680130"} Jul 17 17:25:58.290680 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.290539 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" event={"ID":"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b","Type":"ContainerStarted","Data":"9c8347d69d5353c34cc85a68491d4cc72d6c226d8c73c2997b90913d8babe299"} Jul 17 17:25:58.290680 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.290580 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" event={"ID":"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b","Type":"ContainerStarted","Data":"0defa7c30fcafc5990fd657477a32975f6e8985377c9dec4f63e348a3cd533f0"} Jul 17 17:25:58.291746 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.291717 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gt246" event={"ID":"f6a2178f-b9d1-48d5-963d-49d79be84b3b","Type":"ContainerStarted","Data":"c0c9b8ddc9a527643fd63821e21f9e701c130def2f896f9fde6ea0e51a05da85"} Jul 17 17:25:58.293347 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.293207 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" event={"ID":"8a63170d-0b4c-466d-8171-8e08637042d5","Type":"ContainerStarted","Data":"9e4c91e3bae0da700b65d0b52be6546921e67f1a9c27528add2fe6aeaa39b0ae"} Jul 17 17:25:58.311895 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.311825 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:58.338191 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.338159 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.338310 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.338210 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.338310 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.338241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.338310 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.338281 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.338482 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.338398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.338482 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.338458 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.338580 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.338490 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.338580 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.338525 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-config-volume\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.338580 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.338556 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zggwt\" (UniqueName: \"kubernetes.io/projected/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-kube-api-access-zggwt\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.338735 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.338586 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-config-out\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.338735 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.338610 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.338735 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.338660 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.338735 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.338697 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-web-config\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.339328 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.339294 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.339841 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.339814 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.340515 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.340489 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.342264 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.342167 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.342264 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.342218 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.342428 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.342326 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.346674 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.342359 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.346674 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.343452 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-config-volume\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.346674 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.345366 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-web-config\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.346998 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.346978 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.347678 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.347241 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.349554 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.349535 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-config-out\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.352415 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.352387 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zggwt\" (UniqueName: \"kubernetes.io/projected/cccb9aa5-4d7e-460f-a66c-885c50c0fd61-kube-api-access-zggwt\") pod \"alertmanager-main-0\" (UID: \"cccb9aa5-4d7e-460f-a66c-885c50c0fd61\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.412105 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.412073 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:25:58.433323 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.433257 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:25:58.726737 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:58.726660 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:25:58.952492 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:25:58.952456 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccb9aa5_4d7e_460f_a66c_885c50c0fd61.slice/crio-c693b06044ba1bd660a8a56c6c4620571831d85363bc6c1a6cfb6a662ed8bd15 WatchSource:0}: Error finding container c693b06044ba1bd660a8a56c6c4620571831d85363bc6c1a6cfb6a662ed8bd15: Status 404 returned error can't find the container with id c693b06044ba1bd660a8a56c6c4620571831d85363bc6c1a6cfb6a662ed8bd15 Jul 17 17:25:59.300810 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:59.300762 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gt246" event={"ID":"f6a2178f-b9d1-48d5-963d-49d79be84b3b","Type":"ContainerStarted","Data":"91168b25a9f05fbc536cd01e3fbd8f2708462acbf80e7c200d0cf07350cfe488"} Jul 17 17:25:59.303442 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:59.303404 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" event={"ID":"8a63170d-0b4c-466d-8171-8e08637042d5","Type":"ContainerStarted","Data":"e63925acbea352de8b58f4dcf52479bd752a5c27b3b91fd126045805190a4795"} Jul 17 17:25:59.303581 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:59.303469 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" event={"ID":"8a63170d-0b4c-466d-8171-8e08637042d5","Type":"ContainerStarted","Data":"471242953cf44bfe0a347944bd74d2383698005cbc8c4da0dc764c0d568c2ec9"} Jul 17 17:25:59.303581 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:59.303551 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" event={"ID":"8a63170d-0b4c-466d-8171-8e08637042d5","Type":"ContainerStarted","Data":"0c4c57ec58bfcb80ca25befa1516457f5ed0484b55e533f7666f20f3f8051494"} Jul 17 17:25:59.304966 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:59.304939 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cccb9aa5-4d7e-460f-a66c-885c50c0fd61","Type":"ContainerStarted","Data":"c693b06044ba1bd660a8a56c6c4620571831d85363bc6c1a6cfb6a662ed8bd15"} Jul 17 17:25:59.358407 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:25:59.358362 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-tb5v5" podStartSLOduration=0.919954821 podStartE2EDuration="2.358347647s" totalImagesPullingTime="1.438392826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:59.35668443 +0000 UTC m=+80.092108472" watchObservedRunningTime="2026-07-17 17:25:59.358347647 +0000 UTC m=+80.093771693" Jul 17 17:26:00.025895 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.024446 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq"] Jul 17 17:26:00.046436 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.039824 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.046436 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.045047 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 17 17:26:00.046436 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.045381 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lqpn8\"" Jul 17 17:26:00.046436 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.045509 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 17 17:26:00.046436 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.045838 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-drcpknaef6rau\"" Jul 17 17:26:00.050144 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.049050 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq"] Jul 17 17:26:00.051989 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.051020 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 17 17:26:00.051989 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.051278 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 17 17:26:00.051989 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.051602 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 17 17:26:00.057258 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.057230 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:00.120810 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.120773 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:00.156970 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.156936 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.157169 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.156982 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.157169 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.157039 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.157169 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.157073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzmjn\" (UniqueName: \"kubernetes.io/projected/3963c290-c171-49e5-a957-a902e5ea7c46-kube-api-access-dzmjn\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.157169 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.157100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.157169 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.157129 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-grpc-tls\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.157430 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.157177 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3963c290-c171-49e5-a957-a902e5ea7c46-metrics-client-ca\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.157430 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.157255 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-thanos-querier-tls\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.258352 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.258326 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-thanos-querier-tls\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.258549 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.258377 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.258549 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.258401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.258549 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.258438 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.258549 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.258476 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzmjn\" (UniqueName: \"kubernetes.io/projected/3963c290-c171-49e5-a957-a902e5ea7c46-kube-api-access-dzmjn\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.258549 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.258507 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.258549 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.258543 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-grpc-tls\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.258811 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.258599 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3963c290-c171-49e5-a957-a902e5ea7c46-metrics-client-ca\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.259648 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.259541 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3963c290-c171-49e5-a957-a902e5ea7c46-metrics-client-ca\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.261027 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.261006 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.261163 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.261139 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.261367 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.261347 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-thanos-querier-tls\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.261654 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.261605 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.261837 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.261818 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-grpc-tls\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.261933 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.261871 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3963c290-c171-49e5-a957-a902e5ea7c46-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.266173 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.266147 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzmjn\" (UniqueName: \"kubernetes.io/projected/3963c290-c171-49e5-a957-a902e5ea7c46-kube-api-access-dzmjn\") pod \"thanos-querier-86cdc6ff4c-zdvsq\" (UID: \"3963c290-c171-49e5-a957-a902e5ea7c46\") " pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.310717 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.310686 2573 generic.go:356] "Generic (PLEG): container finished" podID="cccb9aa5-4d7e-460f-a66c-885c50c0fd61" containerID="f7542902924f4c7ae4908e3e708f57303ee2d4916705f5f17f32f567317c35b4" exitCode=0 Jul 17 17:26:00.310845 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.310736 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cccb9aa5-4d7e-460f-a66c-885c50c0fd61","Type":"ContainerDied","Data":"f7542902924f4c7ae4908e3e708f57303ee2d4916705f5f17f32f567317c35b4"} Jul 17 17:26:00.312994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.312973 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" event={"ID":"04577b57-1d9c-49f0-bee5-55cbb5cf8d9b","Type":"ContainerStarted","Data":"6db0c43923f3cc52870efc840c77fdd71151b3a032890ac0b3b24416bd4d4b59"} Jul 17 17:26:00.314368 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.314349 2573 generic.go:356] "Generic (PLEG): container finished" podID="f6a2178f-b9d1-48d5-963d-49d79be84b3b" containerID="91168b25a9f05fbc536cd01e3fbd8f2708462acbf80e7c200d0cf07350cfe488" exitCode=0 Jul 17 17:26:00.314450 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.314391 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gt246" event={"ID":"f6a2178f-b9d1-48d5-963d-49d79be84b3b","Type":"ContainerDied","Data":"91168b25a9f05fbc536cd01e3fbd8f2708462acbf80e7c200d0cf07350cfe488"} Jul 17 17:26:00.329269 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.329248 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:00.360661 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.360571 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:00.380928 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.380887 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wnqgx" podStartSLOduration=3.175074737 podStartE2EDuration="4.380870672s" totalImagesPullingTime="1.205795935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:25:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:26:00.380021556 +0000 UTC m=+81.115445608" watchObservedRunningTime="2026-07-17 17:26:00.380870672 +0000 UTC m=+81.116294730" Jul 17 17:26:00.430406 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.430381 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:00.503673 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:00.503652 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq"] Jul 17 17:26:00.505364 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:26:00.505343 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3963c290_c171_49e5_a957_a902e5ea7c46.slice/crio-65088ce5669b6f3ff652dfcb8934e344ff662adf412b6ee20a4e050a16a544db WatchSource:0}: Error finding container 65088ce5669b6f3ff652dfcb8934e344ff662adf412b6ee20a4e050a16a544db: Status 404 returned error can't find the container with id 65088ce5669b6f3ff652dfcb8934e344ff662adf412b6ee20a4e050a16a544db Jul 17 17:26:01.320483 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.320366 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gt246" event={"ID":"f6a2178f-b9d1-48d5-963d-49d79be84b3b","Type":"ContainerStarted","Data":"7be295c20a498435cc054763d34a5f4f91e5e8135607fd237f7a90e7094c681b"} Jul 17 17:26:01.320483 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.320433 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gt246" event={"ID":"f6a2178f-b9d1-48d5-963d-49d79be84b3b","Type":"ContainerStarted","Data":"d24b042a37a662d597d9e148245c75fc5032fe15f7c302e6d97693567f7ef9b9"} Jul 17 17:26:01.321613 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.321592 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" event={"ID":"3963c290-c171-49e5-a957-a902e5ea7c46","Type":"ContainerStarted","Data":"65088ce5669b6f3ff652dfcb8934e344ff662adf412b6ee20a4e050a16a544db"} Jul 17 17:26:01.344735 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.344688 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gt246" podStartSLOduration=2.720052411 podStartE2EDuration="4.344664904s" totalImagesPullingTime="1.524646528s" totalInitContainerRuntime="99.965965ms" isStatefulPod=true podCreationTimestamp="2026-07-17 17:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:26:01.34370827 +0000 UTC m=+82.079132318" watchObservedRunningTime="2026-07-17 17:26:01.344664904 +0000 UTC m=+82.080088995" Jul 17 17:26:01.460861 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.460799 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-677bc7dc54-nvxff"] Jul 17 17:26:01.464606 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.464583 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.468510 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.468361 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 17 17:26:01.468510 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.468377 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 17 17:26:01.468510 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.468466 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5ldtnuveckq9d\"" Jul 17 17:26:01.468781 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.468644 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-sqg46\"" Jul 17 17:26:01.469735 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.469713 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 17 17:26:01.469840 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.469762 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 17 17:26:01.478314 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.478290 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-677bc7dc54-nvxff"] Jul 17 17:26:01.571098 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.571009 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e9120bc9-f36e-49b2-9477-9133a8c995b5-secret-metrics-server-tls\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.571098 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.571058 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e9120bc9-f36e-49b2-9477-9133a8c995b5-metrics-server-audit-profiles\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.571098 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.571095 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e9120bc9-f36e-49b2-9477-9133a8c995b5-audit-log\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.571326 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.571157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9120bc9-f36e-49b2-9477-9133a8c995b5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.571326 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.571237 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e9120bc9-f36e-49b2-9477-9133a8c995b5-secret-metrics-server-client-certs\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.571326 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.571267 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx56h\" (UniqueName: \"kubernetes.io/projected/e9120bc9-f36e-49b2-9477-9133a8c995b5-kube-api-access-xx56h\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.571326 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.571302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e9120bc9-f36e-49b2-9477-9133a8c995b5-client-ca-bundle\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.673023 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.672643 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e9120bc9-f36e-49b2-9477-9133a8c995b5-secret-metrics-server-client-certs\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.673023 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.672717 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx56h\" (UniqueName: \"kubernetes.io/projected/e9120bc9-f36e-49b2-9477-9133a8c995b5-kube-api-access-xx56h\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.673023 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.672767 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e9120bc9-f36e-49b2-9477-9133a8c995b5-client-ca-bundle\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.673023 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.672827 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e9120bc9-f36e-49b2-9477-9133a8c995b5-secret-metrics-server-tls\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.673023 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.672858 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e9120bc9-f36e-49b2-9477-9133a8c995b5-metrics-server-audit-profiles\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.673023 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.672895 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e9120bc9-f36e-49b2-9477-9133a8c995b5-audit-log\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.673023 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.672925 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9120bc9-f36e-49b2-9477-9133a8c995b5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.673999 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.673947 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e9120bc9-f36e-49b2-9477-9133a8c995b5-audit-log\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.674121 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.674050 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9120bc9-f36e-49b2-9477-9133a8c995b5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.675847 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.675818 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e9120bc9-f36e-49b2-9477-9133a8c995b5-secret-metrics-server-client-certs\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.676107 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.675977 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e9120bc9-f36e-49b2-9477-9133a8c995b5-metrics-server-audit-profiles\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.676418 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.676373 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e9120bc9-f36e-49b2-9477-9133a8c995b5-client-ca-bundle\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.677149 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.677105 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e9120bc9-f36e-49b2-9477-9133a8c995b5-secret-metrics-server-tls\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.681978 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.681957 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx56h\" (UniqueName: \"kubernetes.io/projected/e9120bc9-f36e-49b2-9477-9133a8c995b5-kube-api-access-xx56h\") pod \"metrics-server-677bc7dc54-nvxff\" (UID: \"e9120bc9-f36e-49b2-9477-9133a8c995b5\") " pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.777294 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.777258 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:01.779506 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.779475 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q5t65"] Jul 17 17:26:01.779506 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.779505 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:01.784051 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.783951 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q5t65" Jul 17 17:26:01.786470 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.786446 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mldp2\"" Jul 17 17:26:01.786577 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.786502 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 17 17:26:01.791917 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.791391 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59745b49cf-5k78f"] Jul 17 17:26:01.796031 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.795926 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q5t65"] Jul 17 17:26:01.796126 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.796036 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.817441 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.817366 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59745b49cf-5k78f"] Jul 17 17:26:01.875091 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.874930 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5vjj\" (UniqueName: \"kubernetes.io/projected/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-kube-api-access-g5vjj\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.875091 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.874979 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-oauth-serving-cert\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.875091 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.875027 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5f12077a-e598-4040-a6f5-494f82567e3e-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-q5t65\" (UID: \"5f12077a-e598-4040-a6f5-494f82567e3e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q5t65" Jul 17 17:26:01.875360 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.875103 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-trusted-ca-bundle\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.875360 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.875138 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-serving-cert\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.875360 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.875188 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dknl\" (UniqueName: \"kubernetes.io/projected/5f12077a-e598-4040-a6f5-494f82567e3e-kube-api-access-6dknl\") pod \"monitoring-plugin-7b7b7bc8bf-q5t65\" (UID: \"5f12077a-e598-4040-a6f5-494f82567e3e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q5t65" Jul 17 17:26:01.875360 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.875211 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-oauth-config\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.875360 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.875249 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-config\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.875360 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.875291 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-service-ca\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.878401 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.878312 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:01.940364 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.940332 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-677bc7dc54-nvxff"] Jul 17 17:26:01.946799 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.946773 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:01.976033 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.976005 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-trusted-ca-bundle\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.976193 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.976063 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-serving-cert\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.976193 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.976112 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dknl\" (UniqueName: \"kubernetes.io/projected/5f12077a-e598-4040-a6f5-494f82567e3e-kube-api-access-6dknl\") pod \"monitoring-plugin-7b7b7bc8bf-q5t65\" (UID: \"5f12077a-e598-4040-a6f5-494f82567e3e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q5t65" Jul 17 17:26:01.976193 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.976137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-oauth-config\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.976193 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.976162 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-config\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.976403 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.976202 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-service-ca\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.976403 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.976247 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5vjj\" (UniqueName: \"kubernetes.io/projected/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-kube-api-access-g5vjj\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.976403 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.976278 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-oauth-serving-cert\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.976403 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.976322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5f12077a-e598-4040-a6f5-494f82567e3e-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-q5t65\" (UID: \"5f12077a-e598-4040-a6f5-494f82567e3e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q5t65" Jul 17 17:26:01.977554 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.976963 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-trusted-ca-bundle\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.977554 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.977431 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-config\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.977554 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.977547 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-service-ca\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.978127 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.978100 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-oauth-serving-cert\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.979545 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.979523 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5f12077a-e598-4040-a6f5-494f82567e3e-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-q5t65\" (UID: \"5f12077a-e598-4040-a6f5-494f82567e3e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q5t65" Jul 17 17:26:01.980026 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.979982 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-serving-cert\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.980171 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.980144 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-oauth-config\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:01.992339 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.992312 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dknl\" (UniqueName: \"kubernetes.io/projected/5f12077a-e598-4040-a6f5-494f82567e3e-kube-api-access-6dknl\") pod \"monitoring-plugin-7b7b7bc8bf-q5t65\" (UID: \"5f12077a-e598-4040-a6f5-494f82567e3e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q5t65" Jul 17 17:26:01.992565 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:01.992544 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5vjj\" (UniqueName: \"kubernetes.io/projected/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-kube-api-access-g5vjj\") pod \"console-59745b49cf-5k78f\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:02.014061 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:02.013997 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:02.123327 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:02.123238 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q5t65" Jul 17 17:26:02.132492 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:02.130120 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:02.133485 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:02.133463 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:02.230712 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:02.230674 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:02.725249 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:26:02.725218 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9120bc9_f36e_49b2_9477_9133a8c995b5.slice/crio-bbc1ebde4f9f993021cd1d2adb544f1a06bc85665fa42b413bd7f76b44b20e86 WatchSource:0}: Error finding container bbc1ebde4f9f993021cd1d2adb544f1a06bc85665fa42b413bd7f76b44b20e86: Status 404 returned error can't find the container with id bbc1ebde4f9f993021cd1d2adb544f1a06bc85665fa42b413bd7f76b44b20e86 Jul 17 17:26:02.943651 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:02.943578 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q5t65"] Jul 17 17:26:02.948964 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:26:02.948933 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f12077a_e598_4040_a6f5_494f82567e3e.slice/crio-7cea7665bb07072b8bbcbbc924c0f54025b25345e0ffa4fa1981c9fff8240b97 WatchSource:0}: Error finding container 7cea7665bb07072b8bbcbbc924c0f54025b25345e0ffa4fa1981c9fff8240b97: Status 404 returned error can't find the container with id 7cea7665bb07072b8bbcbbc924c0f54025b25345e0ffa4fa1981c9fff8240b97 Jul 17 17:26:02.966723 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:02.966671 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59745b49cf-5k78f"] Jul 17 17:26:02.971314 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:26:02.971283 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab444b5_0b48_4a21_8f99_aef8eeb1dbcb.slice/crio-ee86a6a3d77038447eb5093cbe1a5a1f5990b91f407192217ae117727197f972 WatchSource:0}: Error finding container ee86a6a3d77038447eb5093cbe1a5a1f5990b91f407192217ae117727197f972: Status 404 returned error can't find the container with id ee86a6a3d77038447eb5093cbe1a5a1f5990b91f407192217ae117727197f972 Jul 17 17:26:03.333604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.333559 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q5t65" event={"ID":"5f12077a-e598-4040-a6f5-494f82567e3e","Type":"ContainerStarted","Data":"7cea7665bb07072b8bbcbbc924c0f54025b25345e0ffa4fa1981c9fff8240b97"} Jul 17 17:26:03.335472 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.335438 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59745b49cf-5k78f" event={"ID":"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb","Type":"ContainerStarted","Data":"00eb5a600d34fb1b5f6b9ea7978a388e097d47e7903595d63be23f655e262fdf"} Jul 17 17:26:03.335636 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.335484 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59745b49cf-5k78f" event={"ID":"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb","Type":"ContainerStarted","Data":"ee86a6a3d77038447eb5093cbe1a5a1f5990b91f407192217ae117727197f972"} Jul 17 17:26:03.339223 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.339189 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cccb9aa5-4d7e-460f-a66c-885c50c0fd61","Type":"ContainerStarted","Data":"b0a50b3a2bb15ef6bea4c610eabad530459549e2a078305f21c406ae813b692c"} Jul 17 17:26:03.339491 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.339256 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cccb9aa5-4d7e-460f-a66c-885c50c0fd61","Type":"ContainerStarted","Data":"ca6d7f6f8ee4be5d017a06a6b73d3f895b16a4ff0063f0e949abc103627d525e"} Jul 17 17:26:03.339491 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.339295 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cccb9aa5-4d7e-460f-a66c-885c50c0fd61","Type":"ContainerStarted","Data":"6b595898471a894d8e0ff09f1e022cd0c3b8d9c38156f5b6e0581380610f9176"} Jul 17 17:26:03.339491 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.339339 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cccb9aa5-4d7e-460f-a66c-885c50c0fd61","Type":"ContainerStarted","Data":"ddc37927a55e812284c67d73cbd68ab6440c022142ebf6628f19b6cd4589fa35"} Jul 17 17:26:03.339491 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.339377 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cccb9aa5-4d7e-460f-a66c-885c50c0fd61","Type":"ContainerStarted","Data":"d8d72fc71ab3fa919ff819c2626e369418a59ca69da5ebf5f50a4d683ee6a11b"} Jul 17 17:26:03.341300 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.341266 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" event={"ID":"e9120bc9-f36e-49b2-9477-9133a8c995b5","Type":"ContainerStarted","Data":"bbc1ebde4f9f993021cd1d2adb544f1a06bc85665fa42b413bd7f76b44b20e86"} Jul 17 17:26:03.344391 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.344367 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" event={"ID":"3963c290-c171-49e5-a957-a902e5ea7c46","Type":"ContainerStarted","Data":"ee4c3c907462683bbf8b12373a3c5a8653dc10a90f3af9bd0aa06eb74b1db967"} Jul 17 17:26:03.344494 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.344435 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" event={"ID":"3963c290-c171-49e5-a957-a902e5ea7c46","Type":"ContainerStarted","Data":"211ad039c203e1c44c4beb328db00d9505787cdd39cece09bb3e9eb432f9f9e1"} Jul 17 17:26:03.344494 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.344477 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" event={"ID":"3963c290-c171-49e5-a957-a902e5ea7c46","Type":"ContainerStarted","Data":"b4b26e19488b968603d4f224155b9e75a954466fc2b95610f6eda9ffc90fdac2"} Jul 17 17:26:03.367772 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.367718 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59745b49cf-5k78f" podStartSLOduration=2.367699131 podStartE2EDuration="2.367699131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:26:03.36530851 +0000 UTC m=+84.100732562" watchObservedRunningTime="2026-07-17 17:26:03.367699131 +0000 UTC m=+84.103123181" Jul 17 17:26:03.429641 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.429582 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:26:03.435330 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.435307 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.437175 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.437148 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:03.442199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.441989 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 17:26:03.442199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.442031 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 17:26:03.442199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.442031 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 17:26:03.442199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.442083 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 17:26:03.442199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.442194 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 17:26:03.442494 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.442356 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 17:26:03.442494 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.442435 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 17:26:03.442638 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.442598 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 17:26:03.443437 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.443193 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 17:26:03.443437 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.443368 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 17:26:03.443579 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.443488 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 17:26:03.443579 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.443528 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 17:26:03.443579 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.443571 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-du3p846obtl2d\"" Jul 17 17:26:03.443579 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.443400 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k4sr9\"" Jul 17 17:26:03.443803 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.443306 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 17:26:03.445147 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.445126 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 17:26:03.465554 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.465522 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:26:03.488745 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.488704 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.488920 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.488760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-config\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.488920 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.488785 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.488920 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.488816 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.488920 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.488846 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.488920 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.488917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.489202 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.488948 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.489202 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.488975 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.489202 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.489000 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.489202 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.489063 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.489202 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.489095 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.489546 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.489178 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.489546 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.489281 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.489546 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.489317 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.489546 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.489345 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.489546 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.489419 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46csb\" (UniqueName: \"kubernetes.io/projected/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-kube-api-access-46csb\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.489546 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.489456 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.489546 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.489491 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.489546 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.489518 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.489546 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.489543 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.590745 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.590796 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.590822 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.590847 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.590871 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.590894 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.590935 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46csb\" (UniqueName: \"kubernetes.io/projected/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-kube-api-access-46csb\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.590973 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.591012 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.591044 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.591070 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.591117 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.591144 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-config\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.591166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.591189 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.591213 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.592231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.591273 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.593176 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.591296 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.593176 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.591326 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.593176 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.591353 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.593176 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.592922 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.594858 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.593842 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.594858 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.594121 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.594858 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.594673 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.610758 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.610367 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.614015 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.612906 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.615010 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.614859 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.616468 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.616443 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.618645 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.618388 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.630434 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.630409 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.631131 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.631076 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.631710 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.631655 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-config\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.632430 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.632269 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.633410 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.633356 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.634269 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.633966 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.635042 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.634985 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.635505 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.635380 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46csb\" (UniqueName: \"kubernetes.io/projected/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-kube-api-access-46csb\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.636036 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.636014 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.636127 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.636032 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.640372 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.640330 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c987b6d-0ea4-48a1-85ae-1ece97c3de0b-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.751701 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.751660 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:03.928522 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:03.926312 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:26:04.049190 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:26:04.049152 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c987b6d_0ea4_48a1_85ae_1ece97c3de0b.slice/crio-ec1854e53b23ff39a91ab11fcf80acb7c4855813ebe4c78ab92846ac5c961d84 WatchSource:0}: Error finding container ec1854e53b23ff39a91ab11fcf80acb7c4855813ebe4c78ab92846ac5c961d84: Status 404 returned error can't find the container with id ec1854e53b23ff39a91ab11fcf80acb7c4855813ebe4c78ab92846ac5c961d84 Jul 17 17:26:04.350677 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:04.350645 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b","Type":"ContainerStarted","Data":"ec1854e53b23ff39a91ab11fcf80acb7c4855813ebe4c78ab92846ac5c961d84"} Jul 17 17:26:04.377029 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:04.376994 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:04.480032 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:04.479992 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:05.262357 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:05.262281 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:05.360515 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:05.360472 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" event={"ID":"e9120bc9-f36e-49b2-9477-9133a8c995b5","Type":"ContainerStarted","Data":"0fdbcfa4db8845f766ba034f72df911c84494895ae4605c5b90cc06a5ee09418"} Jul 17 17:26:05.361913 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:05.361880 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:05.363733 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:05.363651 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" event={"ID":"3963c290-c171-49e5-a957-a902e5ea7c46","Type":"ContainerStarted","Data":"9df76ea6504c181d0ac0a99717c576454fd1c64735c721b3caacec780c0d9165"} Jul 17 17:26:05.365451 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:05.365427 2573 generic.go:356] "Generic (PLEG): container finished" podID="7c987b6d-0ea4-48a1-85ae-1ece97c3de0b" containerID="ea1a77fc92a0cbdbe9fba8c6f6548eee935c668b36de11bc712cd43221d2c527" exitCode=0 Jul 17 17:26:05.365578 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:05.365553 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b","Type":"ContainerDied","Data":"ea1a77fc92a0cbdbe9fba8c6f6548eee935c668b36de11bc712cd43221d2c527"} Jul 17 17:26:05.366987 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:05.366961 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q5t65" event={"ID":"5f12077a-e598-4040-a6f5-494f82567e3e","Type":"ContainerStarted","Data":"72a653d7e917b5763fab51a080377fd308477866e43e781f41e1c055abca2c2d"} Jul 17 17:26:05.367294 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:05.367245 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q5t65" Jul 17 17:26:05.371864 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:05.371777 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cccb9aa5-4d7e-460f-a66c-885c50c0fd61","Type":"ContainerStarted","Data":"ed463901e5048ec9681797e6b467ffd4cb05fc9d6a1c0a252ff273e9da9c0b40"} Jul 17 17:26:05.374533 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:05.374517 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q5t65" Jul 17 17:26:05.378120 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:05.378084 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" podStartSLOduration=1.97512576 podStartE2EDuration="4.378070504s" totalImagesPullingTime="2.402944744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:26:05.377649929 +0000 UTC m=+86.113073979" watchObservedRunningTime="2026-07-17 17:26:05.378070504 +0000 UTC m=+86.113494550" Jul 17 17:26:05.451358 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:05.450355 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q5t65" podStartSLOduration=2.224106688 podStartE2EDuration="4.450336205s" totalImagesPullingTime="2.226229517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:26:05.449429185 +0000 UTC m=+86.184853238" watchObservedRunningTime="2026-07-17 17:26:05.450336205 +0000 UTC m=+86.185760255" Jul 17 17:26:05.451358 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:05.451108 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.6645417409999999 podStartE2EDuration="7.451095951s" totalImagesPullingTime="5.771327745s" totalInitContainerRuntime="15.226465ms" isStatefulPod=false podCreationTimestamp="2026-07-17 17:25:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:26:05.433301509 +0000 UTC m=+86.168725560" watchObservedRunningTime="2026-07-17 17:26:05.451095951 +0000 UTC m=+86.186520001" Jul 17 17:26:06.380848 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:06.380805 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" event={"ID":"3963c290-c171-49e5-a957-a902e5ea7c46","Type":"ContainerStarted","Data":"fea15d3d130ae640ec97e6c7e35b722035ac9ef2f8334534b2c327e979d55156"} Jul 17 17:26:06.381282 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:06.380893 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" event={"ID":"3963c290-c171-49e5-a957-a902e5ea7c46","Type":"ContainerStarted","Data":"b34a5e96772333a17207eaee6d5c5f6570b93353f4730011d0d4c2647f3239d2"} Jul 17 17:26:06.381601 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:06.381573 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:06.404391 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:06.404347 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" podStartSLOduration=2.968555773 podStartE2EDuration="7.404332056s" totalImagesPullingTime="4.435776283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:25:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:26:06.402573616 +0000 UTC m=+87.137997686" watchObservedRunningTime="2026-07-17 17:26:06.404332056 +0000 UTC m=+87.139756105" Jul 17 17:26:07.391256 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:07.391229 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86cdc6ff4c-zdvsq" Jul 17 17:26:09.374866 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:09.374831 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:09.399360 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:09.399330 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b","Type":"ContainerStarted","Data":"65806b60407230518038dab000f758a0159369fe264c91502763ce90b3e2b4bf"} Jul 17 17:26:09.399640 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:09.399596 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b","Type":"ContainerStarted","Data":"39b39d1b423f001ddd47367ee21d0babd2a397d0a31b23f4ab0390f88a8f2d0a"} Jul 17 17:26:09.456530 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:09.456493 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:10.405699 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:10.405665 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b","Type":"ContainerStarted","Data":"7f192240408db7acc4159a1fad7d270166e1b03521d3bd0c57b2f83a5cdadad4"} Jul 17 17:26:10.406060 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:10.405753 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b","Type":"ContainerStarted","Data":"04256f08a390d4bc5208a62ad185c808ed3ba98da5756a466123961ee4649ab8"} Jul 17 17:26:10.406060 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:10.405779 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b","Type":"ContainerStarted","Data":"c55443b7269ba31cbb8253d47574555f7c6d4a17331bd61359fc48e8ac7ca413"} Jul 17 17:26:10.406060 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:10.405802 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c987b6d-0ea4-48a1-85ae-1ece97c3de0b","Type":"ContainerStarted","Data":"812c758297f51b819c6966b33aa80d9576af35ddfa7313eb8892573122c637da"} Jul 17 17:26:10.434539 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:10.434447 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.667817141 podStartE2EDuration="7.434427539s" totalImagesPullingTime="3.746760147s" totalInitContainerRuntime="19.850251ms" isStatefulPod=false podCreationTimestamp="2026-07-17 17:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:26:10.432811699 +0000 UTC m=+91.168235770" watchObservedRunningTime="2026-07-17 17:26:10.434427539 +0000 UTC m=+91.169851589" Jul 17 17:26:12.131432 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:12.131396 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:12.131827 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:12.131458 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:12.136306 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:12.136285 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:12.418355 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:12.418279 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:26:12.465717 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:12.465686 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76f6c85f7-cs49h"] Jul 17 17:26:12.606595 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:12.606562 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:12.700115 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:12.700016 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:12.800523 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:12.800496 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:13.752469 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:13.752434 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:26:14.329064 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:14.329010 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-94cdd4948-wj9zl" podUID="78c15898-e760-4e54-adc0-e13c7fa473a9" containerName="console" containerID="cri-o://7f7ff0e39230fc737ee98fc4843026a62253757c278e71a74efb90a27f4dbe22" gracePeriod=15 Jul 17 17:26:15.068545 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.068523 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:26:15.202973 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.202901 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-oauth-serving-cert\") pod \"78c15898-e760-4e54-adc0-e13c7fa473a9\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " Jul 17 17:26:15.202973 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.202935 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-service-ca\") pod \"78c15898-e760-4e54-adc0-e13c7fa473a9\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " Jul 17 17:26:15.203144 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.202978 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78c15898-e760-4e54-adc0-e13c7fa473a9-console-serving-cert\") pod \"78c15898-e760-4e54-adc0-e13c7fa473a9\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " Jul 17 17:26:15.203144 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.202995 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-console-config\") pod \"78c15898-e760-4e54-adc0-e13c7fa473a9\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " Jul 17 17:26:15.203144 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.203022 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxbhb\" (UniqueName: \"kubernetes.io/projected/78c15898-e760-4e54-adc0-e13c7fa473a9-kube-api-access-zxbhb\") pod \"78c15898-e760-4e54-adc0-e13c7fa473a9\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " Jul 17 17:26:15.203144 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.203076 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78c15898-e760-4e54-adc0-e13c7fa473a9-console-oauth-config\") pod \"78c15898-e760-4e54-adc0-e13c7fa473a9\" (UID: \"78c15898-e760-4e54-adc0-e13c7fa473a9\") " Jul 17 17:26:15.203411 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.203384 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "78c15898-e760-4e54-adc0-e13c7fa473a9" (UID: "78c15898-e760-4e54-adc0-e13c7fa473a9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:26:15.203483 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.203417 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-service-ca" (OuterVolumeSpecName: "service-ca") pod "78c15898-e760-4e54-adc0-e13c7fa473a9" (UID: "78c15898-e760-4e54-adc0-e13c7fa473a9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:26:15.203483 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.203423 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-console-config" (OuterVolumeSpecName: "console-config") pod "78c15898-e760-4e54-adc0-e13c7fa473a9" (UID: "78c15898-e760-4e54-adc0-e13c7fa473a9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:26:15.205538 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.205516 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c15898-e760-4e54-adc0-e13c7fa473a9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "78c15898-e760-4e54-adc0-e13c7fa473a9" (UID: "78c15898-e760-4e54-adc0-e13c7fa473a9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:26:15.205538 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.205521 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c15898-e760-4e54-adc0-e13c7fa473a9-kube-api-access-zxbhb" (OuterVolumeSpecName: "kube-api-access-zxbhb") pod "78c15898-e760-4e54-adc0-e13c7fa473a9" (UID: "78c15898-e760-4e54-adc0-e13c7fa473a9"). InnerVolumeSpecName "kube-api-access-zxbhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:26:15.205735 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.205577 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c15898-e760-4e54-adc0-e13c7fa473a9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "78c15898-e760-4e54-adc0-e13c7fa473a9" (UID: "78c15898-e760-4e54-adc0-e13c7fa473a9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:26:15.303612 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.303578 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:26:15.303612 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.303610 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78c15898-e760-4e54-adc0-e13c7fa473a9-console-serving-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:26:15.303612 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.303636 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-console-config\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:26:15.303797 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.303647 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxbhb\" (UniqueName: \"kubernetes.io/projected/78c15898-e760-4e54-adc0-e13c7fa473a9-kube-api-access-zxbhb\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:26:15.303797 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.303656 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78c15898-e760-4e54-adc0-e13c7fa473a9-console-oauth-config\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:26:15.303797 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.303665 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78c15898-e760-4e54-adc0-e13c7fa473a9-oauth-serving-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:26:15.428932 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.428904 2573 generic.go:356] "Generic (PLEG): container finished" podID="78c15898-e760-4e54-adc0-e13c7fa473a9" containerID="7f7ff0e39230fc737ee98fc4843026a62253757c278e71a74efb90a27f4dbe22" exitCode=0 Jul 17 17:26:15.429073 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.428948 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-94cdd4948-wj9zl" Jul 17 17:26:15.429073 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.428981 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-94cdd4948-wj9zl" event={"ID":"78c15898-e760-4e54-adc0-e13c7fa473a9","Type":"ContainerDied","Data":"7f7ff0e39230fc737ee98fc4843026a62253757c278e71a74efb90a27f4dbe22"} Jul 17 17:26:15.429073 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.429030 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-94cdd4948-wj9zl" event={"ID":"78c15898-e760-4e54-adc0-e13c7fa473a9","Type":"ContainerDied","Data":"0a2752cf23b5f7645d58ee9f14f7887dc9336431f10a79e83e4e533e6fd0b72f"} Jul 17 17:26:15.429073 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.429050 2573 scope.go:117] "RemoveContainer" containerID="7f7ff0e39230fc737ee98fc4843026a62253757c278e71a74efb90a27f4dbe22" Jul 17 17:26:15.438477 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.438462 2573 scope.go:117] "RemoveContainer" containerID="7f7ff0e39230fc737ee98fc4843026a62253757c278e71a74efb90a27f4dbe22" Jul 17 17:26:15.438773 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:26:15.438756 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f7ff0e39230fc737ee98fc4843026a62253757c278e71a74efb90a27f4dbe22\": container with ID starting with 7f7ff0e39230fc737ee98fc4843026a62253757c278e71a74efb90a27f4dbe22 not found: ID does not exist" containerID="7f7ff0e39230fc737ee98fc4843026a62253757c278e71a74efb90a27f4dbe22" Jul 17 17:26:15.438834 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.438781 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f7ff0e39230fc737ee98fc4843026a62253757c278e71a74efb90a27f4dbe22"} err="failed to get container status \"7f7ff0e39230fc737ee98fc4843026a62253757c278e71a74efb90a27f4dbe22\": rpc error: code = NotFound desc = could not find container \"7f7ff0e39230fc737ee98fc4843026a62253757c278e71a74efb90a27f4dbe22\": container with ID starting with 7f7ff0e39230fc737ee98fc4843026a62253757c278e71a74efb90a27f4dbe22 not found: ID does not exist" Jul 17 17:26:15.449274 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.449250 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-94cdd4948-wj9zl"] Jul 17 17:26:15.454101 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.454048 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-94cdd4948-wj9zl"] Jul 17 17:26:15.776541 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:15.776505 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78c15898-e760-4e54-adc0-e13c7fa473a9" path="/var/lib/kubelet/pods/78c15898-e760-4e54-adc0-e13c7fa473a9/volumes" Jul 17 17:26:21.262163 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:21.262136 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pvv46" Jul 17 17:26:21.779946 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:21.779917 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:21.780131 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:21.779966 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:21.836168 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:21.836137 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:22.152199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:22.152132 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sstpk_66a066f3-5b00-49a6-8b43-7498b829ca1e/serve-healthcheck-canary/0.log" Jul 17 17:26:24.352684 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:24.352652 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:24.408898 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:24.408870 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:36.370167 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:36.370136 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:36.471192 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:36.471152 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:36.521208 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:36.521164 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:26:37.485735 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.485696 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76f6c85f7-cs49h" podUID="458425f9-4fdd-453b-8b14-0fafcb83185f" containerName="console" containerID="cri-o://10fa84f945d288d8771f216a00db87e7b02e3db6bae6e89546bade05a8161641" gracePeriod=15 Jul 17 17:26:37.726235 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.726206 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:26:37.809913 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.809846 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-console-config\") pod \"458425f9-4fdd-453b-8b14-0fafcb83185f\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " Jul 17 17:26:37.809913 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.809911 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/458425f9-4fdd-453b-8b14-0fafcb83185f-console-oauth-config\") pod \"458425f9-4fdd-453b-8b14-0fafcb83185f\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " Jul 17 17:26:37.810074 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.809968 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-oauth-serving-cert\") pod \"458425f9-4fdd-453b-8b14-0fafcb83185f\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " Jul 17 17:26:37.810074 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.810004 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n496f\" (UniqueName: \"kubernetes.io/projected/458425f9-4fdd-453b-8b14-0fafcb83185f-kube-api-access-n496f\") pod \"458425f9-4fdd-453b-8b14-0fafcb83185f\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " Jul 17 17:26:37.810074 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.810034 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/458425f9-4fdd-453b-8b14-0fafcb83185f-console-serving-cert\") pod \"458425f9-4fdd-453b-8b14-0fafcb83185f\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " Jul 17 17:26:37.810074 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.810063 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-trusted-ca-bundle\") pod \"458425f9-4fdd-453b-8b14-0fafcb83185f\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " Jul 17 17:26:37.810256 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.810103 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-service-ca\") pod \"458425f9-4fdd-453b-8b14-0fafcb83185f\" (UID: \"458425f9-4fdd-453b-8b14-0fafcb83185f\") " Jul 17 17:26:37.810310 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.810266 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-console-config" (OuterVolumeSpecName: "console-config") pod "458425f9-4fdd-453b-8b14-0fafcb83185f" (UID: "458425f9-4fdd-453b-8b14-0fafcb83185f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:26:37.810612 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.810360 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "458425f9-4fdd-453b-8b14-0fafcb83185f" (UID: "458425f9-4fdd-453b-8b14-0fafcb83185f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:26:37.810612 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.810387 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-console-config\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:26:37.810612 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.810557 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "458425f9-4fdd-453b-8b14-0fafcb83185f" (UID: "458425f9-4fdd-453b-8b14-0fafcb83185f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:26:37.810869 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.810676 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-service-ca" (OuterVolumeSpecName: "service-ca") pod "458425f9-4fdd-453b-8b14-0fafcb83185f" (UID: "458425f9-4fdd-453b-8b14-0fafcb83185f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:26:37.812310 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.812288 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/458425f9-4fdd-453b-8b14-0fafcb83185f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "458425f9-4fdd-453b-8b14-0fafcb83185f" (UID: "458425f9-4fdd-453b-8b14-0fafcb83185f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:26:37.812611 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.812590 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/458425f9-4fdd-453b-8b14-0fafcb83185f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "458425f9-4fdd-453b-8b14-0fafcb83185f" (UID: "458425f9-4fdd-453b-8b14-0fafcb83185f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:26:37.812719 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.812604 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458425f9-4fdd-453b-8b14-0fafcb83185f-kube-api-access-n496f" (OuterVolumeSpecName: "kube-api-access-n496f") pod "458425f9-4fdd-453b-8b14-0fafcb83185f" (UID: "458425f9-4fdd-453b-8b14-0fafcb83185f"). InnerVolumeSpecName "kube-api-access-n496f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:26:37.911270 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.911236 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n496f\" (UniqueName: \"kubernetes.io/projected/458425f9-4fdd-453b-8b14-0fafcb83185f-kube-api-access-n496f\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:26:37.911270 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.911264 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/458425f9-4fdd-453b-8b14-0fafcb83185f-console-serving-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:26:37.911270 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.911274 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-trusted-ca-bundle\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:26:37.911485 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.911282 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:26:37.911485 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.911291 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/458425f9-4fdd-453b-8b14-0fafcb83185f-console-oauth-config\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:26:37.911485 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:37.911300 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/458425f9-4fdd-453b-8b14-0fafcb83185f-oauth-serving-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:26:38.532664 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:38.532615 2573 generic.go:356] "Generic (PLEG): container finished" podID="458425f9-4fdd-453b-8b14-0fafcb83185f" containerID="10fa84f945d288d8771f216a00db87e7b02e3db6bae6e89546bade05a8161641" exitCode=0 Jul 17 17:26:38.533089 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:38.532681 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f6c85f7-cs49h" Jul 17 17:26:38.533089 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:38.532694 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f6c85f7-cs49h" event={"ID":"458425f9-4fdd-453b-8b14-0fafcb83185f","Type":"ContainerDied","Data":"10fa84f945d288d8771f216a00db87e7b02e3db6bae6e89546bade05a8161641"} Jul 17 17:26:38.533089 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:38.532754 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f6c85f7-cs49h" event={"ID":"458425f9-4fdd-453b-8b14-0fafcb83185f","Type":"ContainerDied","Data":"0fb9e04c3a55e20ac9b278f2aa4c75772908aa2f0a241741144c45f2eb2b8e9c"} Jul 17 17:26:38.533089 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:38.532782 2573 scope.go:117] "RemoveContainer" containerID="10fa84f945d288d8771f216a00db87e7b02e3db6bae6e89546bade05a8161641" Jul 17 17:26:38.544358 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:38.544334 2573 scope.go:117] "RemoveContainer" containerID="10fa84f945d288d8771f216a00db87e7b02e3db6bae6e89546bade05a8161641" Jul 17 17:26:38.545078 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:26:38.545054 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10fa84f945d288d8771f216a00db87e7b02e3db6bae6e89546bade05a8161641\": container with ID starting with 10fa84f945d288d8771f216a00db87e7b02e3db6bae6e89546bade05a8161641 not found: ID does not exist" containerID="10fa84f945d288d8771f216a00db87e7b02e3db6bae6e89546bade05a8161641" Jul 17 17:26:38.545181 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:38.545087 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10fa84f945d288d8771f216a00db87e7b02e3db6bae6e89546bade05a8161641"} err="failed to get container status \"10fa84f945d288d8771f216a00db87e7b02e3db6bae6e89546bade05a8161641\": rpc error: code = NotFound desc = could not find container \"10fa84f945d288d8771f216a00db87e7b02e3db6bae6e89546bade05a8161641\": container with ID starting with 10fa84f945d288d8771f216a00db87e7b02e3db6bae6e89546bade05a8161641 not found: ID does not exist" Jul 17 17:26:38.556804 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:38.556778 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76f6c85f7-cs49h"] Jul 17 17:26:38.562720 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:38.562696 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76f6c85f7-cs49h"] Jul 17 17:26:39.777078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:39.777044 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="458425f9-4fdd-453b-8b14-0fafcb83185f" path="/var/lib/kubelet/pods/458425f9-4fdd-453b-8b14-0fafcb83185f/volumes" Jul 17 17:26:41.784930 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:41.784901 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:26:41.788662 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:26:41.788640 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-677bc7dc54-nvxff" Jul 17 17:27:03.752870 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:03.752834 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:27:03.776580 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:03.776545 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:27:04.648332 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:04.648301 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:27:16.415635 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:16.415578 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:16.684696 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:16.684576 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:21.525868 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.525828 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7fb58646b7-2srz5"] Jul 17 17:27:21.526235 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.526167 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78c15898-e760-4e54-adc0-e13c7fa473a9" containerName="console" Jul 17 17:27:21.526235 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.526178 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="78c15898-e760-4e54-adc0-e13c7fa473a9" containerName="console" Jul 17 17:27:21.526235 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.526196 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="458425f9-4fdd-453b-8b14-0fafcb83185f" containerName="console" Jul 17 17:27:21.526235 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.526203 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="458425f9-4fdd-453b-8b14-0fafcb83185f" containerName="console" Jul 17 17:27:21.526368 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.526258 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="458425f9-4fdd-453b-8b14-0fafcb83185f" containerName="console" Jul 17 17:27:21.526368 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.526265 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="78c15898-e760-4e54-adc0-e13c7fa473a9" containerName="console" Jul 17 17:27:21.530640 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.530606 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.533519 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.533500 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-z4gsj\"" Jul 17 17:27:21.533519 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.533509 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 17 17:27:21.533827 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.533780 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 17 17:27:21.533935 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.533916 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 17 17:27:21.534040 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.534008 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 17 17:27:21.534158 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.534145 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 17 17:27:21.539906 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.539884 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 17 17:27:21.544459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.544436 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fb58646b7-2srz5"] Jul 17 17:27:21.592543 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.592508 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbd1bee5-1088-4975-af1e-9357c3b142ff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.592730 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.592553 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbd1bee5-1088-4975-af1e-9357c3b142ff-metrics-client-ca\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.592730 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.592586 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbd1bee5-1088-4975-af1e-9357c3b142ff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.592730 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.592673 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbd1bee5-1088-4975-af1e-9357c3b142ff-serving-certs-ca-bundle\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.592730 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.592705 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bbd1bee5-1088-4975-af1e-9357c3b142ff-secret-telemeter-client\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.592730 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.592723 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bbd1bee5-1088-4975-af1e-9357c3b142ff-telemeter-client-tls\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.592893 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.592804 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bbd1bee5-1088-4975-af1e-9357c3b142ff-federate-client-tls\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.592893 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.592821 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fwzg\" (UniqueName: \"kubernetes.io/projected/bbd1bee5-1088-4975-af1e-9357c3b142ff-kube-api-access-8fwzg\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.693952 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.693913 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbd1bee5-1088-4975-af1e-9357c3b142ff-metrics-client-ca\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.694133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.693967 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbd1bee5-1088-4975-af1e-9357c3b142ff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.694133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.693997 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbd1bee5-1088-4975-af1e-9357c3b142ff-serving-certs-ca-bundle\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.694133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.694014 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bbd1bee5-1088-4975-af1e-9357c3b142ff-secret-telemeter-client\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.694133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.694029 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bbd1bee5-1088-4975-af1e-9357c3b142ff-telemeter-client-tls\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.694133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.694062 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bbd1bee5-1088-4975-af1e-9357c3b142ff-federate-client-tls\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.694133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.694077 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fwzg\" (UniqueName: \"kubernetes.io/projected/bbd1bee5-1088-4975-af1e-9357c3b142ff-kube-api-access-8fwzg\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.694133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.694119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbd1bee5-1088-4975-af1e-9357c3b142ff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.694743 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.694715 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbd1bee5-1088-4975-af1e-9357c3b142ff-metrics-client-ca\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.694860 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.694798 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbd1bee5-1088-4975-af1e-9357c3b142ff-serving-certs-ca-bundle\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.695227 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.695201 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbd1bee5-1088-4975-af1e-9357c3b142ff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.696785 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.696756 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbd1bee5-1088-4975-af1e-9357c3b142ff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.696865 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.696784 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bbd1bee5-1088-4975-af1e-9357c3b142ff-secret-telemeter-client\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.696961 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.696945 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bbd1bee5-1088-4975-af1e-9357c3b142ff-telemeter-client-tls\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.696998 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.696985 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bbd1bee5-1088-4975-af1e-9357c3b142ff-federate-client-tls\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.703027 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.703001 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fwzg\" (UniqueName: \"kubernetes.io/projected/bbd1bee5-1088-4975-af1e-9357c3b142ff-kube-api-access-8fwzg\") pod \"telemeter-client-7fb58646b7-2srz5\" (UID: \"bbd1bee5-1088-4975-af1e-9357c3b142ff\") " pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.842386 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.842312 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" Jul 17 17:27:21.972426 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:21.972398 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fb58646b7-2srz5"] Jul 17 17:27:21.974370 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:27:21.974343 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd1bee5_1088_4975_af1e_9357c3b142ff.slice/crio-789294caa4f4201ff6a4af14373d6a7306b5f445c779d3b583ebaa00090b3a5f WatchSource:0}: Error finding container 789294caa4f4201ff6a4af14373d6a7306b5f445c779d3b583ebaa00090b3a5f: Status 404 returned error can't find the container with id 789294caa4f4201ff6a4af14373d6a7306b5f445c779d3b583ebaa00090b3a5f Jul 17 17:27:22.709678 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:22.709641 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" event={"ID":"bbd1bee5-1088-4975-af1e-9357c3b142ff","Type":"ContainerStarted","Data":"789294caa4f4201ff6a4af14373d6a7306b5f445c779d3b583ebaa00090b3a5f"} Jul 17 17:27:24.719798 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:24.719766 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" event={"ID":"bbd1bee5-1088-4975-af1e-9357c3b142ff","Type":"ContainerStarted","Data":"a85abaf301e8ab17e0e2ad027434ba3e71eca572fd3ee83cfcbda06a38e409c4"} Jul 17 17:27:24.720174 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:24.719824 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" event={"ID":"bbd1bee5-1088-4975-af1e-9357c3b142ff","Type":"ContainerStarted","Data":"f2f1720e004c94c856378c9ca430f8f07e86998f3e2e1ca57558b2d082122b4a"} Jul 17 17:27:24.720174 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:24.719852 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" event={"ID":"bbd1bee5-1088-4975-af1e-9357c3b142ff","Type":"ContainerStarted","Data":"67e85db9d8c434bbe5886357ce62ca72408ee0305448bec3588d95e3bc381d1a"} Jul 17 17:27:24.741715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:24.741647 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7fb58646b7-2srz5" podStartSLOduration=1.7968467609999998 podStartE2EDuration="3.741615409s" totalImagesPullingTime="1.944768648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:27:24.740590757 +0000 UTC m=+165.476014825" watchObservedRunningTime="2026-07-17 17:27:24.741615409 +0000 UTC m=+165.477039451" Jul 17 17:27:25.535169 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.535138 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cbc98858c-h7v5h"] Jul 17 17:27:25.542839 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.542807 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.549737 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.549707 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cbc98858c-h7v5h"] Jul 17 17:27:25.593530 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.593497 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:25.632368 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.632338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql29h\" (UniqueName: \"kubernetes.io/projected/e3b1280a-0af4-4463-8f21-7cb797c79aa0-kube-api-access-ql29h\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.632368 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.632378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-config\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.632578 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.632398 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-service-ca\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.632578 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.632463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-trusted-ca-bundle\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.632704 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.632583 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-oauth-config\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.632704 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.632676 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-serving-cert\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.632704 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.632692 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-oauth-serving-cert\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.733794 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.733756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-oauth-config\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.734237 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.733807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-serving-cert\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.734237 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.733824 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-oauth-serving-cert\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.734237 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.733853 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql29h\" (UniqueName: \"kubernetes.io/projected/e3b1280a-0af4-4463-8f21-7cb797c79aa0-kube-api-access-ql29h\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.734237 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.733881 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-config\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.734237 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.733902 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-service-ca\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.734237 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.733940 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-trusted-ca-bundle\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.734774 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.734744 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-oauth-serving-cert\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.734896 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.734860 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-service-ca\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.734896 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.734882 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-trusted-ca-bundle\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.735009 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.734883 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-config\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.736545 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.736501 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-oauth-config\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.738088 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.738064 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-serving-cert\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.742400 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.742374 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql29h\" (UniqueName: \"kubernetes.io/projected/e3b1280a-0af4-4463-8f21-7cb797c79aa0-kube-api-access-ql29h\") pod \"console-cbc98858c-h7v5h\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.795044 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.794957 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:25.854701 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.854670 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:25.899101 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.899066 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:26.000005 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:25.999979 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cbc98858c-h7v5h"] Jul 17 17:27:26.002894 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:27:26.002866 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b1280a_0af4_4463_8f21_7cb797c79aa0.slice/crio-f6a551fcfa3bff44d0335098e5187663bdc6bb029cf6c52ee02bda674d75114d WatchSource:0}: Error finding container f6a551fcfa3bff44d0335098e5187663bdc6bb029cf6c52ee02bda674d75114d: Status 404 returned error can't find the container with id f6a551fcfa3bff44d0335098e5187663bdc6bb029cf6c52ee02bda674d75114d Jul 17 17:27:26.340133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:26.340015 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:26.439890 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:26.439861 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:26.730878 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:26.730794 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbc98858c-h7v5h" event={"ID":"e3b1280a-0af4-4463-8f21-7cb797c79aa0","Type":"ContainerStarted","Data":"b6604ab054f086f618967ee4c7bcb0e2faad2e5ce78fd43ebf4c1ad748995bfd"} Jul 17 17:27:26.730878 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:26.730849 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbc98858c-h7v5h" event={"ID":"e3b1280a-0af4-4463-8f21-7cb797c79aa0","Type":"ContainerStarted","Data":"f6a551fcfa3bff44d0335098e5187663bdc6bb029cf6c52ee02bda674d75114d"} Jul 17 17:27:26.749008 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:26.748959 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cbc98858c-h7v5h" podStartSLOduration=1.748944862 podStartE2EDuration="1.748944862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:27:26.748085956 +0000 UTC m=+167.483510007" watchObservedRunningTime="2026-07-17 17:27:26.748944862 +0000 UTC m=+167.484368910" Jul 17 17:27:31.673440 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:31.673357 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:31.734009 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:31.733948 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:34.120017 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.119981 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cbc98858c-h7v5h"] Jul 17 17:27:34.150713 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.150684 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7697fff5c8-6fzct"] Jul 17 17:27:34.153665 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.153648 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.164645 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.163926 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7697fff5c8-6fzct"] Jul 17 17:27:34.213875 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.213845 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27d51da0-37a5-419c-b605-cff572bbbbed-console-oauth-config\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.213875 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.213877 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-trusted-ca-bundle\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.214099 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.213900 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-service-ca\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.214099 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.214045 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf6fd\" (UniqueName: \"kubernetes.io/projected/27d51da0-37a5-419c-b605-cff572bbbbed-kube-api-access-mf6fd\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.214099 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.214089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-oauth-serving-cert\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.214257 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.214150 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-console-config\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.214257 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.214181 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27d51da0-37a5-419c-b605-cff572bbbbed-console-serving-cert\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.315097 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.315063 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27d51da0-37a5-419c-b605-cff572bbbbed-console-serving-cert\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.315261 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.315129 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27d51da0-37a5-419c-b605-cff572bbbbed-console-oauth-config\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.315261 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.315152 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-trusted-ca-bundle\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.315261 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.315183 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-service-ca\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.315261 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.315251 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf6fd\" (UniqueName: \"kubernetes.io/projected/27d51da0-37a5-419c-b605-cff572bbbbed-kube-api-access-mf6fd\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.315466 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.315282 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-oauth-serving-cert\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.315466 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.315316 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-console-config\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.316034 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.316009 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-console-config\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.316134 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.316054 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-trusted-ca-bundle\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.316134 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.316084 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-service-ca\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.316134 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.316120 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-oauth-serving-cert\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.318010 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.317986 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27d51da0-37a5-419c-b605-cff572bbbbed-console-oauth-config\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.318100 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.318083 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27d51da0-37a5-419c-b605-cff572bbbbed-console-serving-cert\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.323154 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.323135 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf6fd\" (UniqueName: \"kubernetes.io/projected/27d51da0-37a5-419c-b605-cff572bbbbed-kube-api-access-mf6fd\") pod \"console-7697fff5c8-6fzct\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.468091 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.468021 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:34.596936 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.596912 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7697fff5c8-6fzct"] Jul 17 17:27:34.599034 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:27:34.599004 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d51da0_37a5_419c_b605_cff572bbbbed.slice/crio-485349c24902fc4a8084d4c42b0050cb011745e42285a83339fcc866d93390cc WatchSource:0}: Error finding container 485349c24902fc4a8084d4c42b0050cb011745e42285a83339fcc866d93390cc: Status 404 returned error can't find the container with id 485349c24902fc4a8084d4c42b0050cb011745e42285a83339fcc866d93390cc Jul 17 17:27:34.770363 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.770330 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7697fff5c8-6fzct" event={"ID":"27d51da0-37a5-419c-b605-cff572bbbbed","Type":"ContainerStarted","Data":"3c14a406e3aa9c3d2b3448b421ab7890a7e4728820421500db9ec61264a051d8"} Jul 17 17:27:34.770525 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.770381 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7697fff5c8-6fzct" event={"ID":"27d51da0-37a5-419c-b605-cff572bbbbed","Type":"ContainerStarted","Data":"485349c24902fc4a8084d4c42b0050cb011745e42285a83339fcc866d93390cc"} Jul 17 17:27:34.789149 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:34.789102 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7697fff5c8-6fzct" podStartSLOduration=0.789085728 podStartE2EDuration="789.085728ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:27:34.788289119 +0000 UTC m=+175.523713195" watchObservedRunningTime="2026-07-17 17:27:34.789085728 +0000 UTC m=+175.524509777" Jul 17 17:27:35.854953 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:35.854919 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:43.697204 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:43.697164 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:43.746138 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:43.746106 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:44.469104 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:44.469069 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:44.469289 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:44.469118 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:44.473832 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:44.473809 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:44.813445 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:44.813415 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:27:44.859122 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:44.859092 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59745b49cf-5k78f"] Jul 17 17:27:44.966389 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:44.966346 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:45.066428 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:45.066336 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:45.117337 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:45.117307 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:52.396298 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:52.396256 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:52.447059 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:52.447014 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:54.356421 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:54.356387 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:54.454659 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:54.454607 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:27:59.139751 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.139709 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cbc98858c-h7v5h" podUID="e3b1280a-0af4-4463-8f21-7cb797c79aa0" containerName="console" containerID="cri-o://b6604ab054f086f618967ee4c7bcb0e2faad2e5ce78fd43ebf4c1ad748995bfd" gracePeriod=15 Jul 17 17:27:59.376466 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.376444 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:59.445570 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.445487 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-oauth-config\") pod \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " Jul 17 17:27:59.445570 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.445528 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-config\") pod \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " Jul 17 17:27:59.445570 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.445567 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-service-ca\") pod \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " Jul 17 17:27:59.445859 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.445589 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-trusted-ca-bundle\") pod \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " Jul 17 17:27:59.445859 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.445665 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-serving-cert\") pod \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " Jul 17 17:27:59.445859 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.445693 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql29h\" (UniqueName: \"kubernetes.io/projected/e3b1280a-0af4-4463-8f21-7cb797c79aa0-kube-api-access-ql29h\") pod \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " Jul 17 17:27:59.445859 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.445714 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-oauth-serving-cert\") pod \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\" (UID: \"e3b1280a-0af4-4463-8f21-7cb797c79aa0\") " Jul 17 17:27:59.446058 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.445999 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-config" (OuterVolumeSpecName: "console-config") pod "e3b1280a-0af4-4463-8f21-7cb797c79aa0" (UID: "e3b1280a-0af4-4463-8f21-7cb797c79aa0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:27:59.446113 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.446063 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-service-ca" (OuterVolumeSpecName: "service-ca") pod "e3b1280a-0af4-4463-8f21-7cb797c79aa0" (UID: "e3b1280a-0af4-4463-8f21-7cb797c79aa0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:27:59.446233 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.446209 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e3b1280a-0af4-4463-8f21-7cb797c79aa0" (UID: "e3b1280a-0af4-4463-8f21-7cb797c79aa0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:27:59.446295 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.446244 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e3b1280a-0af4-4463-8f21-7cb797c79aa0" (UID: "e3b1280a-0af4-4463-8f21-7cb797c79aa0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:27:59.447802 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.447782 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e3b1280a-0af4-4463-8f21-7cb797c79aa0" (UID: "e3b1280a-0af4-4463-8f21-7cb797c79aa0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:27:59.448139 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.448116 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e3b1280a-0af4-4463-8f21-7cb797c79aa0" (UID: "e3b1280a-0af4-4463-8f21-7cb797c79aa0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:27:59.448199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.448116 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b1280a-0af4-4463-8f21-7cb797c79aa0-kube-api-access-ql29h" (OuterVolumeSpecName: "kube-api-access-ql29h") pod "e3b1280a-0af4-4463-8f21-7cb797c79aa0" (UID: "e3b1280a-0af4-4463-8f21-7cb797c79aa0"). InnerVolumeSpecName "kube-api-access-ql29h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:27:59.546566 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.546530 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:27:59.546566 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.546560 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-trusted-ca-bundle\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:27:59.546566 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.546571 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-serving-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:27:59.546808 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.546581 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql29h\" (UniqueName: \"kubernetes.io/projected/e3b1280a-0af4-4463-8f21-7cb797c79aa0-kube-api-access-ql29h\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:27:59.546808 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.546590 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-oauth-serving-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:27:59.546808 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.546599 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-oauth-config\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:27:59.546808 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.546608 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3b1280a-0af4-4463-8f21-7cb797c79aa0-console-config\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:27:59.874848 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.874793 2573 generic.go:356] "Generic (PLEG): container finished" podID="e3b1280a-0af4-4463-8f21-7cb797c79aa0" containerID="b6604ab054f086f618967ee4c7bcb0e2faad2e5ce78fd43ebf4c1ad748995bfd" exitCode=0 Jul 17 17:27:59.874848 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.874836 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbc98858c-h7v5h" Jul 17 17:27:59.874848 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.874849 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbc98858c-h7v5h" event={"ID":"e3b1280a-0af4-4463-8f21-7cb797c79aa0","Type":"ContainerDied","Data":"b6604ab054f086f618967ee4c7bcb0e2faad2e5ce78fd43ebf4c1ad748995bfd"} Jul 17 17:27:59.875092 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.874889 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbc98858c-h7v5h" event={"ID":"e3b1280a-0af4-4463-8f21-7cb797c79aa0","Type":"ContainerDied","Data":"f6a551fcfa3bff44d0335098e5187663bdc6bb029cf6c52ee02bda674d75114d"} Jul 17 17:27:59.875092 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.874909 2573 scope.go:117] "RemoveContainer" containerID="b6604ab054f086f618967ee4c7bcb0e2faad2e5ce78fd43ebf4c1ad748995bfd" Jul 17 17:27:59.883048 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.883030 2573 scope.go:117] "RemoveContainer" containerID="b6604ab054f086f618967ee4c7bcb0e2faad2e5ce78fd43ebf4c1ad748995bfd" Jul 17 17:27:59.883306 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:27:59.883284 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6604ab054f086f618967ee4c7bcb0e2faad2e5ce78fd43ebf4c1ad748995bfd\": container with ID starting with b6604ab054f086f618967ee4c7bcb0e2faad2e5ce78fd43ebf4c1ad748995bfd not found: ID does not exist" containerID="b6604ab054f086f618967ee4c7bcb0e2faad2e5ce78fd43ebf4c1ad748995bfd" Jul 17 17:27:59.883359 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.883310 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6604ab054f086f618967ee4c7bcb0e2faad2e5ce78fd43ebf4c1ad748995bfd"} err="failed to get container status \"b6604ab054f086f618967ee4c7bcb0e2faad2e5ce78fd43ebf4c1ad748995bfd\": rpc error: code = NotFound desc = could not find container \"b6604ab054f086f618967ee4c7bcb0e2faad2e5ce78fd43ebf4c1ad748995bfd\": container with ID starting with b6604ab054f086f618967ee4c7bcb0e2faad2e5ce78fd43ebf4c1ad748995bfd not found: ID does not exist" Jul 17 17:27:59.891467 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.891435 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cbc98858c-h7v5h"] Jul 17 17:27:59.897025 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:27:59.897004 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cbc98858c-h7v5h"] Jul 17 17:28:01.777035 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:01.777002 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3b1280a-0af4-4463-8f21-7cb797c79aa0" path="/var/lib/kubelet/pods/e3b1280a-0af4-4463-8f21-7cb797c79aa0/volumes" Jul 17 17:28:09.880188 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:09.880119 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59745b49cf-5k78f" podUID="0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb" containerName="console" containerID="cri-o://00eb5a600d34fb1b5f6b9ea7978a388e097d47e7903595d63be23f655e262fdf" gracePeriod=15 Jul 17 17:28:10.121926 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.121901 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:28:10.237598 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.237573 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5vjj\" (UniqueName: \"kubernetes.io/projected/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-kube-api-access-g5vjj\") pod \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " Jul 17 17:28:10.237769 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.237615 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-trusted-ca-bundle\") pod \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " Jul 17 17:28:10.237769 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.237698 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-oauth-serving-cert\") pod \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " Jul 17 17:28:10.237769 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.237713 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-service-ca\") pod \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " Jul 17 17:28:10.237769 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.237727 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-config\") pod \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " Jul 17 17:28:10.237769 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.237750 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-oauth-config\") pod \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " Jul 17 17:28:10.238026 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.237789 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-serving-cert\") pod \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\" (UID: \"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb\") " Jul 17 17:28:10.238150 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.238107 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-config" (OuterVolumeSpecName: "console-config") pod "0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb" (UID: "0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:28:10.238150 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.238127 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb" (UID: "0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:28:10.238150 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.238132 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb" (UID: "0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:28:10.238264 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.238122 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-service-ca" (OuterVolumeSpecName: "service-ca") pod "0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb" (UID: "0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:28:10.239748 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.239717 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-kube-api-access-g5vjj" (OuterVolumeSpecName: "kube-api-access-g5vjj") pod "0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb" (UID: "0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb"). InnerVolumeSpecName "kube-api-access-g5vjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:28:10.239877 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.239858 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb" (UID: "0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:28:10.240021 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.240004 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb" (UID: "0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:28:10.338685 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.338651 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-oauth-serving-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:28:10.338685 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.338681 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:28:10.338685 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.338692 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-config\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:28:10.338904 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.338699 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-oauth-config\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:28:10.338904 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.338707 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-console-serving-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:28:10.338904 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.338717 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5vjj\" (UniqueName: \"kubernetes.io/projected/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-kube-api-access-g5vjj\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:28:10.338904 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.338727 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb-trusted-ca-bundle\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:28:10.920545 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.920509 2573 generic.go:356] "Generic (PLEG): container finished" podID="0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb" containerID="00eb5a600d34fb1b5f6b9ea7978a388e097d47e7903595d63be23f655e262fdf" exitCode=0 Jul 17 17:28:10.921032 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.920558 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59745b49cf-5k78f" Jul 17 17:28:10.921032 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.920589 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59745b49cf-5k78f" event={"ID":"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb","Type":"ContainerDied","Data":"00eb5a600d34fb1b5f6b9ea7978a388e097d47e7903595d63be23f655e262fdf"} Jul 17 17:28:10.921032 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.920667 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59745b49cf-5k78f" event={"ID":"0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb","Type":"ContainerDied","Data":"ee86a6a3d77038447eb5093cbe1a5a1f5990b91f407192217ae117727197f972"} Jul 17 17:28:10.921032 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.920689 2573 scope.go:117] "RemoveContainer" containerID="00eb5a600d34fb1b5f6b9ea7978a388e097d47e7903595d63be23f655e262fdf" Jul 17 17:28:10.929424 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.929409 2573 scope.go:117] "RemoveContainer" containerID="00eb5a600d34fb1b5f6b9ea7978a388e097d47e7903595d63be23f655e262fdf" Jul 17 17:28:10.929704 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:28:10.929686 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00eb5a600d34fb1b5f6b9ea7978a388e097d47e7903595d63be23f655e262fdf\": container with ID starting with 00eb5a600d34fb1b5f6b9ea7978a388e097d47e7903595d63be23f655e262fdf not found: ID does not exist" containerID="00eb5a600d34fb1b5f6b9ea7978a388e097d47e7903595d63be23f655e262fdf" Jul 17 17:28:10.929758 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.929713 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00eb5a600d34fb1b5f6b9ea7978a388e097d47e7903595d63be23f655e262fdf"} err="failed to get container status \"00eb5a600d34fb1b5f6b9ea7978a388e097d47e7903595d63be23f655e262fdf\": rpc error: code = NotFound desc = could not find container \"00eb5a600d34fb1b5f6b9ea7978a388e097d47e7903595d63be23f655e262fdf\": container with ID starting with 00eb5a600d34fb1b5f6b9ea7978a388e097d47e7903595d63be23f655e262fdf not found: ID does not exist" Jul 17 17:28:10.941509 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.941486 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59745b49cf-5k78f"] Jul 17 17:28:10.946136 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:10.946117 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59745b49cf-5k78f"] Jul 17 17:28:11.776888 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:11.776852 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb" path="/var/lib/kubelet/pods/0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb/volumes" Jul 17 17:28:38.575977 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:38.575945 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:28:38.672490 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:38.672455 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:28:47.215885 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:47.215851 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:28:47.266002 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:28:47.265970 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:00.136555 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:00.136508 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:00.929332 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:00.929256 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:03.749928 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:03.749895 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:03.848463 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:03.848429 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:06.010666 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:06.010615 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:06.121677 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:06.121647 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:06.243658 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:06.243605 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:06.353017 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:06.352921 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:08.421247 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:08.421215 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:08.875744 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:08.875711 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:10.769719 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:10.769686 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:10.832193 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:10.832149 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:12.778591 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.778550 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2tdjs"] Jul 17 17:29:12.779150 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.779130 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb" containerName="console" Jul 17 17:29:12.779233 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.779153 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb" containerName="console" Jul 17 17:29:12.779233 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.779175 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b1280a-0af4-4463-8f21-7cb797c79aa0" containerName="console" Jul 17 17:29:12.779233 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.779184 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b1280a-0af4-4463-8f21-7cb797c79aa0" containerName="console" Jul 17 17:29:12.779372 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.779279 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b1280a-0af4-4463-8f21-7cb797c79aa0" containerName="console" Jul 17 17:29:12.779372 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.779295 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab444b5-0b48-4a21-8f99-aef8eeb1dbcb" containerName="console" Jul 17 17:29:12.781357 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.781336 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tdjs" Jul 17 17:29:12.783950 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.783934 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 17 17:29:12.784908 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.784888 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 17 17:29:12.784908 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.784905 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-56qxm\"" Jul 17 17:29:12.788825 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.788708 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2tdjs"] Jul 17 17:29:12.888324 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.888284 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzsvc\" (UniqueName: \"kubernetes.io/projected/adbf0561-2117-4f9d-96fd-d13bea4c2f15-kube-api-access-qzsvc\") pod \"cert-manager-cainjector-559cdd8c87-2tdjs\" (UID: \"adbf0561-2117-4f9d-96fd-d13bea4c2f15\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tdjs" Jul 17 17:29:12.888564 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.888410 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adbf0561-2117-4f9d-96fd-d13bea4c2f15-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2tdjs\" (UID: \"adbf0561-2117-4f9d-96fd-d13bea4c2f15\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tdjs" Jul 17 17:29:12.989537 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.989494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzsvc\" (UniqueName: \"kubernetes.io/projected/adbf0561-2117-4f9d-96fd-d13bea4c2f15-kube-api-access-qzsvc\") pod \"cert-manager-cainjector-559cdd8c87-2tdjs\" (UID: \"adbf0561-2117-4f9d-96fd-d13bea4c2f15\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tdjs" Jul 17 17:29:12.989740 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.989608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adbf0561-2117-4f9d-96fd-d13bea4c2f15-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2tdjs\" (UID: \"adbf0561-2117-4f9d-96fd-d13bea4c2f15\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tdjs" Jul 17 17:29:12.997552 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.997520 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adbf0561-2117-4f9d-96fd-d13bea4c2f15-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2tdjs\" (UID: \"adbf0561-2117-4f9d-96fd-d13bea4c2f15\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tdjs" Jul 17 17:29:12.997854 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:12.997834 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzsvc\" (UniqueName: \"kubernetes.io/projected/adbf0561-2117-4f9d-96fd-d13bea4c2f15-kube-api-access-qzsvc\") pod \"cert-manager-cainjector-559cdd8c87-2tdjs\" (UID: \"adbf0561-2117-4f9d-96fd-d13bea4c2f15\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tdjs" Jul 17 17:29:13.102433 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:13.102345 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tdjs" Jul 17 17:29:13.226140 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:13.226112 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2tdjs"] Jul 17 17:29:13.228649 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:29:13.228596 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadbf0561_2117_4f9d_96fd_d13bea4c2f15.slice/crio-7e939ecb78692874b8aaab4f04afbe7ca99314a12bf50634bfe5e5dec513804b WatchSource:0}: Error finding container 7e939ecb78692874b8aaab4f04afbe7ca99314a12bf50634bfe5e5dec513804b: Status 404 returned error can't find the container with id 7e939ecb78692874b8aaab4f04afbe7ca99314a12bf50634bfe5e5dec513804b Jul 17 17:29:14.181652 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:14.181571 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tdjs" event={"ID":"adbf0561-2117-4f9d-96fd-d13bea4c2f15","Type":"ContainerStarted","Data":"7e939ecb78692874b8aaab4f04afbe7ca99314a12bf50634bfe5e5dec513804b"} Jul 17 17:29:16.612477 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:16.612442 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:17.196288 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:17.196252 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tdjs" event={"ID":"adbf0561-2117-4f9d-96fd-d13bea4c2f15","Type":"ContainerStarted","Data":"5d5608638fc9e9e846eed8aa111592e6a49570c365e5d9d7b8b77ecaaa6dd8be"} Jul 17 17:29:17.211123 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:17.211078 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tdjs" podStartSLOduration=1.9165141970000001 podStartE2EDuration="5.211064928s" totalImagesPullingTime="3.294550731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:29:17.210379452 +0000 UTC m=+277.945803525" watchObservedRunningTime="2026-07-17 17:29:17.211064928 +0000 UTC m=+277.946488977" Jul 17 17:29:17.859507 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:17.859463 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:22.349517 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:22.349482 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:22.449645 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:22.449587 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:24.371566 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:24.371534 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jzvxh"] Jul 17 17:29:24.374318 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:24.374297 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jzvxh" Jul 17 17:29:24.376889 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:24.376868 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-nchhc\"" Jul 17 17:29:24.376987 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:24.376902 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 17:29:24.377846 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:24.377829 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 17:29:24.382858 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:24.382839 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jzvxh"] Jul 17 17:29:24.493431 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:24.493397 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25d12de3-1576-4a91-99e6-3b82842116f6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jzvxh\" (UID: \"25d12de3-1576-4a91-99e6-3b82842116f6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jzvxh" Jul 17 17:29:24.493593 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:24.493449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8bp4\" (UniqueName: \"kubernetes.io/projected/25d12de3-1576-4a91-99e6-3b82842116f6-kube-api-access-v8bp4\") pod \"openshift-lws-operator-fd8ccff4c-jzvxh\" (UID: \"25d12de3-1576-4a91-99e6-3b82842116f6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jzvxh" Jul 17 17:29:24.594108 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:24.594068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25d12de3-1576-4a91-99e6-3b82842116f6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jzvxh\" (UID: \"25d12de3-1576-4a91-99e6-3b82842116f6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jzvxh" Jul 17 17:29:24.594295 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:24.594127 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8bp4\" (UniqueName: \"kubernetes.io/projected/25d12de3-1576-4a91-99e6-3b82842116f6-kube-api-access-v8bp4\") pod \"openshift-lws-operator-fd8ccff4c-jzvxh\" (UID: \"25d12de3-1576-4a91-99e6-3b82842116f6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jzvxh" Jul 17 17:29:24.594468 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:24.594446 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25d12de3-1576-4a91-99e6-3b82842116f6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jzvxh\" (UID: \"25d12de3-1576-4a91-99e6-3b82842116f6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jzvxh" Jul 17 17:29:24.610100 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:24.610070 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8bp4\" (UniqueName: \"kubernetes.io/projected/25d12de3-1576-4a91-99e6-3b82842116f6-kube-api-access-v8bp4\") pod \"openshift-lws-operator-fd8ccff4c-jzvxh\" (UID: \"25d12de3-1576-4a91-99e6-3b82842116f6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jzvxh" Jul 17 17:29:24.684277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:24.684184 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jzvxh" Jul 17 17:29:24.807714 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:24.807691 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jzvxh"] Jul 17 17:29:24.810265 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:29:24.810239 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25d12de3_1576_4a91_99e6_3b82842116f6.slice/crio-918c6073f96257a8f2c106f520f9c5add86ffc78b69ce72dcdf230e53cb439e3 WatchSource:0}: Error finding container 918c6073f96257a8f2c106f520f9c5add86ffc78b69ce72dcdf230e53cb439e3: Status 404 returned error can't find the container with id 918c6073f96257a8f2c106f520f9c5add86ffc78b69ce72dcdf230e53cb439e3 Jul 17 17:29:25.227269 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:25.227233 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jzvxh" event={"ID":"25d12de3-1576-4a91-99e6-3b82842116f6","Type":"ContainerStarted","Data":"918c6073f96257a8f2c106f520f9c5add86ffc78b69ce72dcdf230e53cb439e3"} Jul 17 17:29:27.238541 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:27.238505 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jzvxh" event={"ID":"25d12de3-1576-4a91-99e6-3b82842116f6","Type":"ContainerStarted","Data":"10dcf5bcf18beac426a1265d6926bf5c7bec8d546c30032811da19e70e3ef47f"} Jul 17 17:29:27.257595 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:27.257546 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jzvxh" podStartSLOduration=0.967188968 podStartE2EDuration="3.257530852s" totalImagesPullingTime="2.290341884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:29:27.256267145 +0000 UTC m=+287.991691194" watchObservedRunningTime="2026-07-17 17:29:27.257530852 +0000 UTC m=+287.992954900" Jul 17 17:29:27.903474 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:27.903443 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:28.010856 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:28.010805 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:30.592165 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:30.592133 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:39.650572 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:39.650543 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:29:39.656745 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:39.656725 2573 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 17:29:39.656976 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:39.656960 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:29:40.562378 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:40.562343 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:40.617139 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:40.617107 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:44.385849 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:44.385815 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:44.477912 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:44.477878 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:45.038732 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.038702 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:45.270548 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.270516 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:45.370141 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.370039 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:45.422901 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.422868 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8"] Jul 17 17:29:45.474709 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.474679 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8"] Jul 17 17:29:45.474851 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.474776 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.477354 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.477331 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 17:29:45.477571 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.477556 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 17:29:45.477683 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.477639 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-l7hd7\"" Jul 17 17:29:45.477746 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.477682 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 17:29:45.477985 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.477964 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 17:29:45.536514 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.536482 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:45.574005 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.573976 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2c1eaa4-5145-49ca-b9f0-d578671e5bc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cbb795d5f-4qsx8\" (UID: \"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.574125 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.574025 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdflz\" (UniqueName: \"kubernetes.io/projected/a2c1eaa4-5145-49ca-b9f0-d578671e5bc4-kube-api-access-kdflz\") pod \"opendatahub-operator-controller-manager-6cbb795d5f-4qsx8\" (UID: \"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.574125 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.574073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2c1eaa4-5145-49ca-b9f0-d578671e5bc4-tmp\") pod \"opendatahub-operator-controller-manager-6cbb795d5f-4qsx8\" (UID: \"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.574194 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.574150 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2c1eaa4-5145-49ca-b9f0-d578671e5bc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cbb795d5f-4qsx8\" (UID: \"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.574194 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.574185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2c1eaa4-5145-49ca-b9f0-d578671e5bc4-manifests\") pod \"opendatahub-operator-controller-manager-6cbb795d5f-4qsx8\" (UID: \"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.634650 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.634548 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:45.675048 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.675016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2c1eaa4-5145-49ca-b9f0-d578671e5bc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cbb795d5f-4qsx8\" (UID: \"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.675182 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.675069 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2c1eaa4-5145-49ca-b9f0-d578671e5bc4-manifests\") pod \"opendatahub-operator-controller-manager-6cbb795d5f-4qsx8\" (UID: \"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.675182 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.675113 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2c1eaa4-5145-49ca-b9f0-d578671e5bc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cbb795d5f-4qsx8\" (UID: \"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.675182 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.675166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdflz\" (UniqueName: \"kubernetes.io/projected/a2c1eaa4-5145-49ca-b9f0-d578671e5bc4-kube-api-access-kdflz\") pod \"opendatahub-operator-controller-manager-6cbb795d5f-4qsx8\" (UID: \"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.675341 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.675207 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2c1eaa4-5145-49ca-b9f0-d578671e5bc4-tmp\") pod \"opendatahub-operator-controller-manager-6cbb795d5f-4qsx8\" (UID: \"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.675598 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.675571 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2c1eaa4-5145-49ca-b9f0-d578671e5bc4-manifests\") pod \"opendatahub-operator-controller-manager-6cbb795d5f-4qsx8\" (UID: \"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.677321 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.677292 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2c1eaa4-5145-49ca-b9f0-d578671e5bc4-tmp\") pod \"opendatahub-operator-controller-manager-6cbb795d5f-4qsx8\" (UID: \"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.677439 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.677423 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2c1eaa4-5145-49ca-b9f0-d578671e5bc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cbb795d5f-4qsx8\" (UID: \"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.677658 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.677639 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2c1eaa4-5145-49ca-b9f0-d578671e5bc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cbb795d5f-4qsx8\" (UID: \"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.683694 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.683675 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdflz\" (UniqueName: \"kubernetes.io/projected/a2c1eaa4-5145-49ca-b9f0-d578671e5bc4-kube-api-access-kdflz\") pod \"opendatahub-operator-controller-manager-6cbb795d5f-4qsx8\" (UID: \"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.755909 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.755872 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:45.785920 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.785893 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:45.861401 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.861349 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:45.940935 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.940911 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8"] Jul 17 17:29:45.942950 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.942921 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:45.943311 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:29:45.943286 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c1eaa4_5145_49ca_b9f0_d578671e5bc4.slice/crio-9430756274bccc24118fc746cf87048259fb5e927472d3f952e1c424331e03cf WatchSource:0}: Error finding container 9430756274bccc24118fc746cf87048259fb5e927472d3f952e1c424331e03cf: Status 404 returned error can't find the container with id 9430756274bccc24118fc746cf87048259fb5e927472d3f952e1c424331e03cf Jul 17 17:29:45.945094 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:45.945072 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 17:29:46.031066 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:46.031038 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:46.320494 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:46.320459 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" event={"ID":"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4","Type":"ContainerStarted","Data":"9430756274bccc24118fc746cf87048259fb5e927472d3f952e1c424331e03cf"} Jul 17 17:29:46.900260 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:46.900198 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:47.009391 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:47.009357 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:29:49.343162 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:49.343118 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" event={"ID":"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4","Type":"ContainerStarted","Data":"7a36cf613af1d6b6ca1d6d1204e2116375f49fe95270f6bfe2a802e972e5d13d"} Jul 17 17:29:50.349748 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:50.349663 2573 generic.go:356] "Generic (PLEG): container finished" podID="a2c1eaa4-5145-49ca-b9f0-d578671e5bc4" containerID="7a36cf613af1d6b6ca1d6d1204e2116375f49fe95270f6bfe2a802e972e5d13d" exitCode=0 Jul 17 17:29:50.349748 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:50.349706 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" event={"ID":"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4","Type":"ContainerDied","Data":"7a36cf613af1d6b6ca1d6d1204e2116375f49fe95270f6bfe2a802e972e5d13d"} Jul 17 17:29:51.356772 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:51.356735 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" event={"ID":"a2c1eaa4-5145-49ca-b9f0-d578671e5bc4","Type":"ContainerStarted","Data":"0b41224c683560b2715721cc849fea5a2c0d9d70a7257bc42182b11d1a1c5044"} Jul 17 17:29:51.357263 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:51.356871 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:29:51.380910 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:29:51.380858 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" podStartSLOduration=1.970902752 podStartE2EDuration="6.380841501s" totalImagesPullingTime="3.347772657s" totalInitContainerRuntime="1.062166092s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:29:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:29:51.378815862 +0000 UTC m=+312.114239940" watchObservedRunningTime="2026-07-17 17:29:51.380841501 +0000 UTC m=+312.116265549" Jul 17 17:30:02.363306 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:02.363277 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cbb795d5f-4qsx8" Jul 17 17:30:03.297326 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.297293 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k"] Jul 17 17:30:03.300650 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.300612 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" Jul 17 17:30:03.304389 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.304371 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-swl4t\"" Jul 17 17:30:03.304491 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.304373 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 17 17:30:03.304491 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.304407 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 17 17:30:03.304491 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.304449 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 17 17:30:03.310718 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.310699 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k"] Jul 17 17:30:03.437021 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.436988 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ed8fa43-d8c5-4d09-b772-a68f3bbe2804-cert\") pod \"lws-controller-manager-6d77fc5c7f-t227k\" (UID: \"6ed8fa43-d8c5-4d09-b772-a68f3bbe2804\") " pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" Jul 17 17:30:03.437390 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.437030 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6ed8fa43-d8c5-4d09-b772-a68f3bbe2804-manager-config\") pod \"lws-controller-manager-6d77fc5c7f-t227k\" (UID: \"6ed8fa43-d8c5-4d09-b772-a68f3bbe2804\") " pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" Jul 17 17:30:03.437390 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.437048 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ed8fa43-d8c5-4d09-b772-a68f3bbe2804-metrics-cert\") pod \"lws-controller-manager-6d77fc5c7f-t227k\" (UID: \"6ed8fa43-d8c5-4d09-b772-a68f3bbe2804\") " pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" Jul 17 17:30:03.437390 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.437139 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9226k\" (UniqueName: \"kubernetes.io/projected/6ed8fa43-d8c5-4d09-b772-a68f3bbe2804-kube-api-access-9226k\") pod \"lws-controller-manager-6d77fc5c7f-t227k\" (UID: \"6ed8fa43-d8c5-4d09-b772-a68f3bbe2804\") " pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" Jul 17 17:30:03.538331 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.538293 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ed8fa43-d8c5-4d09-b772-a68f3bbe2804-cert\") pod \"lws-controller-manager-6d77fc5c7f-t227k\" (UID: \"6ed8fa43-d8c5-4d09-b772-a68f3bbe2804\") " pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" Jul 17 17:30:03.538497 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.538343 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6ed8fa43-d8c5-4d09-b772-a68f3bbe2804-manager-config\") pod \"lws-controller-manager-6d77fc5c7f-t227k\" (UID: \"6ed8fa43-d8c5-4d09-b772-a68f3bbe2804\") " pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" Jul 17 17:30:03.538497 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.538363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ed8fa43-d8c5-4d09-b772-a68f3bbe2804-metrics-cert\") pod \"lws-controller-manager-6d77fc5c7f-t227k\" (UID: \"6ed8fa43-d8c5-4d09-b772-a68f3bbe2804\") " pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" Jul 17 17:30:03.538497 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.538392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9226k\" (UniqueName: \"kubernetes.io/projected/6ed8fa43-d8c5-4d09-b772-a68f3bbe2804-kube-api-access-9226k\") pod \"lws-controller-manager-6d77fc5c7f-t227k\" (UID: \"6ed8fa43-d8c5-4d09-b772-a68f3bbe2804\") " pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" Jul 17 17:30:03.539120 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.539084 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6ed8fa43-d8c5-4d09-b772-a68f3bbe2804-manager-config\") pod \"lws-controller-manager-6d77fc5c7f-t227k\" (UID: \"6ed8fa43-d8c5-4d09-b772-a68f3bbe2804\") " pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" Jul 17 17:30:03.540931 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.540912 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ed8fa43-d8c5-4d09-b772-a68f3bbe2804-metrics-cert\") pod \"lws-controller-manager-6d77fc5c7f-t227k\" (UID: \"6ed8fa43-d8c5-4d09-b772-a68f3bbe2804\") " pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" Jul 17 17:30:03.540992 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.540945 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ed8fa43-d8c5-4d09-b772-a68f3bbe2804-cert\") pod \"lws-controller-manager-6d77fc5c7f-t227k\" (UID: \"6ed8fa43-d8c5-4d09-b772-a68f3bbe2804\") " pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" Jul 17 17:30:03.546862 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.546841 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9226k\" (UniqueName: \"kubernetes.io/projected/6ed8fa43-d8c5-4d09-b772-a68f3bbe2804-kube-api-access-9226k\") pod \"lws-controller-manager-6d77fc5c7f-t227k\" (UID: \"6ed8fa43-d8c5-4d09-b772-a68f3bbe2804\") " pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" Jul 17 17:30:03.610368 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.610299 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" Jul 17 17:30:03.748593 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.748570 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k"] Jul 17 17:30:03.750933 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:30:03.750906 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed8fa43_d8c5_4d09_b772_a68f3bbe2804.slice/crio-b7b8fb7648fcbe9b5085e293fdde9ea2adc1067c779b73853ea380cff93b67aa WatchSource:0}: Error finding container b7b8fb7648fcbe9b5085e293fdde9ea2adc1067c779b73853ea380cff93b67aa: Status 404 returned error can't find the container with id b7b8fb7648fcbe9b5085e293fdde9ea2adc1067c779b73853ea380cff93b67aa Jul 17 17:30:03.851248 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:03.851216 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:04.011984 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:04.011948 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:04.419090 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:04.419003 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" event={"ID":"6ed8fa43-d8c5-4d09-b772-a68f3bbe2804","Type":"ContainerStarted","Data":"b7b8fb7648fcbe9b5085e293fdde9ea2adc1067c779b73853ea380cff93b67aa"} Jul 17 17:30:08.438093 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:08.438056 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" event={"ID":"6ed8fa43-d8c5-4d09-b772-a68f3bbe2804","Type":"ContainerStarted","Data":"060771d9b05c3a49c23249be0cdd3e1af5b7111875cb4caf6b66327ed3eefec6"} Jul 17 17:30:08.438471 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:08.438248 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" Jul 17 17:30:08.455553 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:08.455506 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" podStartSLOduration=0.995332608 podStartE2EDuration="5.455491396s" totalImagesPullingTime="4.460158788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:30:08.454843132 +0000 UTC m=+329.190267193" watchObservedRunningTime="2026-07-17 17:30:08.455491396 +0000 UTC m=+329.190915445" Jul 17 17:30:09.907104 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:09.907053 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:10.008183 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:10.008152 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:10.108414 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:10.108381 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:10.208760 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:10.208644 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:10.526527 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:10.526494 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:10.961683 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:10.961576 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:14.829229 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:14.829192 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:14.923921 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:14.923885 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d88846968-xdklf"] Jul 17 17:30:14.927524 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:14.927508 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d88846968-xdklf" Jul 17 17:30:14.930773 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:14.930749 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 17 17:30:14.930904 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:14.930805 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nkdjw\"" Jul 17 17:30:14.931875 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:14.931857 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 17:30:14.931927 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:14.931907 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 17 17:30:14.931927 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:14.931920 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 17:30:14.941675 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:14.941652 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d88846968-xdklf"] Jul 17 17:30:15.047585 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:15.047548 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpxs9\" (UniqueName: \"kubernetes.io/projected/e95b5d39-f981-4623-aa3f-762d3a24f3ca-kube-api-access-cpxs9\") pod \"kube-auth-proxy-7d88846968-xdklf\" (UID: \"e95b5d39-f981-4623-aa3f-762d3a24f3ca\") " pod="openshift-ingress/kube-auth-proxy-7d88846968-xdklf" Jul 17 17:30:15.047585 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:15.047590 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e95b5d39-f981-4623-aa3f-762d3a24f3ca-tls-certs\") pod \"kube-auth-proxy-7d88846968-xdklf\" (UID: \"e95b5d39-f981-4623-aa3f-762d3a24f3ca\") " pod="openshift-ingress/kube-auth-proxy-7d88846968-xdklf" Jul 17 17:30:15.047826 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:15.047715 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e95b5d39-f981-4623-aa3f-762d3a24f3ca-tmp\") pod \"kube-auth-proxy-7d88846968-xdklf\" (UID: \"e95b5d39-f981-4623-aa3f-762d3a24f3ca\") " pod="openshift-ingress/kube-auth-proxy-7d88846968-xdklf" Jul 17 17:30:15.148814 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:15.148723 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e95b5d39-f981-4623-aa3f-762d3a24f3ca-tmp\") pod \"kube-auth-proxy-7d88846968-xdklf\" (UID: \"e95b5d39-f981-4623-aa3f-762d3a24f3ca\") " pod="openshift-ingress/kube-auth-proxy-7d88846968-xdklf" Jul 17 17:30:15.148974 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:15.148825 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpxs9\" (UniqueName: \"kubernetes.io/projected/e95b5d39-f981-4623-aa3f-762d3a24f3ca-kube-api-access-cpxs9\") pod \"kube-auth-proxy-7d88846968-xdklf\" (UID: \"e95b5d39-f981-4623-aa3f-762d3a24f3ca\") " pod="openshift-ingress/kube-auth-proxy-7d88846968-xdklf" Jul 17 17:30:15.148974 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:15.148857 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e95b5d39-f981-4623-aa3f-762d3a24f3ca-tls-certs\") pod \"kube-auth-proxy-7d88846968-xdklf\" (UID: \"e95b5d39-f981-4623-aa3f-762d3a24f3ca\") " pod="openshift-ingress/kube-auth-proxy-7d88846968-xdklf" Jul 17 17:30:15.148974 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:30:15.148968 2573 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 17 17:30:15.149139 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:30:15.149048 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e95b5d39-f981-4623-aa3f-762d3a24f3ca-tls-certs podName:e95b5d39-f981-4623-aa3f-762d3a24f3ca nodeName:}" failed. No retries permitted until 2026-07-17 17:30:15.649025776 +0000 UTC m=+336.384449817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e95b5d39-f981-4623-aa3f-762d3a24f3ca-tls-certs") pod "kube-auth-proxy-7d88846968-xdklf" (UID: "e95b5d39-f981-4623-aa3f-762d3a24f3ca") : secret "kube-auth-proxy-tls" not found Jul 17 17:30:15.150959 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:15.150938 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e95b5d39-f981-4623-aa3f-762d3a24f3ca-tmp\") pod \"kube-auth-proxy-7d88846968-xdklf\" (UID: \"e95b5d39-f981-4623-aa3f-762d3a24f3ca\") " pod="openshift-ingress/kube-auth-proxy-7d88846968-xdklf" Jul 17 17:30:15.161255 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:15.161227 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpxs9\" (UniqueName: \"kubernetes.io/projected/e95b5d39-f981-4623-aa3f-762d3a24f3ca-kube-api-access-cpxs9\") pod \"kube-auth-proxy-7d88846968-xdklf\" (UID: \"e95b5d39-f981-4623-aa3f-762d3a24f3ca\") " pod="openshift-ingress/kube-auth-proxy-7d88846968-xdklf" Jul 17 17:30:15.450489 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:15.450413 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:15.653478 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:15.653441 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e95b5d39-f981-4623-aa3f-762d3a24f3ca-tls-certs\") pod \"kube-auth-proxy-7d88846968-xdklf\" (UID: \"e95b5d39-f981-4623-aa3f-762d3a24f3ca\") " pod="openshift-ingress/kube-auth-proxy-7d88846968-xdklf" Jul 17 17:30:15.656030 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:15.656003 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e95b5d39-f981-4623-aa3f-762d3a24f3ca-tls-certs\") pod \"kube-auth-proxy-7d88846968-xdklf\" (UID: \"e95b5d39-f981-4623-aa3f-762d3a24f3ca\") " pod="openshift-ingress/kube-auth-proxy-7d88846968-xdklf" Jul 17 17:30:15.836982 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:15.836948 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d88846968-xdklf" Jul 17 17:30:15.989928 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:15.989894 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d88846968-xdklf"] Jul 17 17:30:15.992903 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:30:15.992877 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95b5d39_f981_4623_aa3f_762d3a24f3ca.slice/crio-048a4cf11510bbad6d3bd68c114e0bf26d5585e7b98bc75a5d2d76f935fcf2c5 WatchSource:0}: Error finding container 048a4cf11510bbad6d3bd68c114e0bf26d5585e7b98bc75a5d2d76f935fcf2c5: Status 404 returned error can't find the container with id 048a4cf11510bbad6d3bd68c114e0bf26d5585e7b98bc75a5d2d76f935fcf2c5 Jul 17 17:30:16.078461 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:16.078426 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:16.155994 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:16.155893 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:16.475885 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:16.475803 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d88846968-xdklf" event={"ID":"e95b5d39-f981-4623-aa3f-762d3a24f3ca","Type":"ContainerStarted","Data":"048a4cf11510bbad6d3bd68c114e0bf26d5585e7b98bc75a5d2d76f935fcf2c5"} Jul 17 17:30:17.573229 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:17.573198 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:19.444788 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:19.444704 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6d77fc5c7f-t227k" Jul 17 17:30:19.493037 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:19.493004 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d88846968-xdklf" event={"ID":"e95b5d39-f981-4623-aa3f-762d3a24f3ca","Type":"ContainerStarted","Data":"8aac6ba60a9dd4d96ea46cdaf4f7976b2c3fbb794884447137a394bb9d2ad205"} Jul 17 17:30:19.550253 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:19.550196 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d88846968-xdklf" podStartSLOduration=2.408455394 podStartE2EDuration="5.55017852s" totalImagesPullingTime="3.141723126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:30:19.546309293 +0000 UTC m=+340.281733343" watchObservedRunningTime="2026-07-17 17:30:19.55017852 +0000 UTC m=+340.285602569" Jul 17 17:30:20.983418 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:20.983383 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:21.186547 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:21.186506 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:21.408413 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:21.408380 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:21.573187 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:21.573155 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:21.677211 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:21.677125 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:21.793005 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:21.792974 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:22.361294 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.361265 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd"] Jul 17 17:30:22.364045 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.364026 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.367800 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.367781 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-hrw6p\"" Jul 17 17:30:22.367964 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.367784 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 17 17:30:22.380355 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.380333 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd"] Jul 17 17:30:22.392578 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.392553 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:22.414679 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.414649 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9606969-27ab-45c3-938c-e661047077c8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.414810 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.414691 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9606969-27ab-45c3-938c-e661047077c8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.414810 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.414761 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9606969-27ab-45c3-938c-e661047077c8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.414904 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.414830 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs9fn\" (UniqueName: \"kubernetes.io/projected/f9606969-27ab-45c3-938c-e661047077c8-kube-api-access-cs9fn\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.414904 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.414867 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9606969-27ab-45c3-938c-e661047077c8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.415010 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.414946 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9606969-27ab-45c3-938c-e661047077c8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.415010 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.414994 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9606969-27ab-45c3-938c-e661047077c8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.415110 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.415023 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9606969-27ab-45c3-938c-e661047077c8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.415110 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.415052 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9606969-27ab-45c3-938c-e661047077c8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.515961 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.515923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9606969-27ab-45c3-938c-e661047077c8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.515961 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.515957 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9606969-27ab-45c3-938c-e661047077c8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.516204 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.515981 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9606969-27ab-45c3-938c-e661047077c8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.516204 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.516005 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9606969-27ab-45c3-938c-e661047077c8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.516204 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.516046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9606969-27ab-45c3-938c-e661047077c8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.516204 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.516080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs9fn\" (UniqueName: \"kubernetes.io/projected/f9606969-27ab-45c3-938c-e661047077c8-kube-api-access-cs9fn\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.516204 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.516110 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9606969-27ab-45c3-938c-e661047077c8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.516204 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.516156 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9606969-27ab-45c3-938c-e661047077c8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.516204 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.516192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9606969-27ab-45c3-938c-e661047077c8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.516586 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.516416 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9606969-27ab-45c3-938c-e661047077c8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.516586 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.516490 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9606969-27ab-45c3-938c-e661047077c8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.516586 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.516553 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9606969-27ab-45c3-938c-e661047077c8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.516751 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.516594 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9606969-27ab-45c3-938c-e661047077c8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.516999 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.516982 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9606969-27ab-45c3-938c-e661047077c8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.518646 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.518599 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9606969-27ab-45c3-938c-e661047077c8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.518750 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.518729 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9606969-27ab-45c3-938c-e661047077c8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.530651 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.530612 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9606969-27ab-45c3-938c-e661047077c8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.535024 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.535004 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs9fn\" (UniqueName: \"kubernetes.io/projected/f9606969-27ab-45c3-938c-e661047077c8-kube-api-access-cs9fn\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45zqbd\" (UID: \"f9606969-27ab-45c3-938c-e661047077c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.646777 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.646692 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:22.674349 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.674324 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:22.807596 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.807564 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:22.829909 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:22.829873 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd"] Jul 17 17:30:22.832456 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:30:22.832424 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9606969_27ab_45c3_938c_e661047077c8.slice/crio-9707d683ce9166174a7c9862542813ced88a076c62714cbffadfa32275599351 WatchSource:0}: Error finding container 9707d683ce9166174a7c9862542813ced88a076c62714cbffadfa32275599351: Status 404 returned error can't find the container with id 9707d683ce9166174a7c9862542813ced88a076c62714cbffadfa32275599351 Jul 17 17:30:23.406583 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:23.406541 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:23.507739 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:23.507699 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:23.512064 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:23.512037 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" event={"ID":"f9606969-27ab-45c3-938c-e661047077c8","Type":"ContainerStarted","Data":"9707d683ce9166174a7c9862542813ced88a076c62714cbffadfa32275599351"} Jul 17 17:30:23.664380 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:23.664255 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:26.548848 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:26.548813 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:30:26.549080 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:26.548892 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:30:26.549080 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:26.548926 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:30:27.531236 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:27.531195 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" event={"ID":"f9606969-27ab-45c3-938c-e661047077c8","Type":"ContainerStarted","Data":"bba1bfee10ea1d42178aa3d83b14390e4bcd7c6514472015a414040e7a928b65"} Jul 17 17:30:27.556798 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:27.556752 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" podStartSLOduration=1.8427785399999999 podStartE2EDuration="5.556736722s" totalImagesPullingTime="3.713958182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:30:27.556414375 +0000 UTC m=+348.291838425" watchObservedRunningTime="2026-07-17 17:30:27.556736722 +0000 UTC m=+348.292160770" Jul 17 17:30:27.674681 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:27.674649 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:27.679082 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:27.679053 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:28.537176 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:28.537150 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:28.537954 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:28.537936 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45zqbd" Jul 17 17:30:34.881245 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:34.881212 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:30:34.943821 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:30:34.943791 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:05.306949 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:05.306912 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:05.454599 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:05.454546 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:05.700131 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:05.700043 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:06.294011 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:06.293972 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:06.483068 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:06.483031 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:15.467219 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:15.467178 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:15.579048 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:15.579010 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:33.148236 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:33.148187 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:36.392675 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:36.392640 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:37.177342 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:37.177307 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:40.047423 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:40.047390 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:40.190237 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:40.190207 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:40.855280 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:40.855235 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:42.055488 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:42.055452 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:43.747971 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:43.747936 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:44.829607 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:44.829565 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:46.707833 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:46.707798 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:46.813042 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:46.813013 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:47.455465 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:47.455434 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:49.800055 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:49.800021 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:50.130068 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:50.129991 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:54.504564 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:54.504528 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:54.805499 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:54.805421 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:55.191773 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:55.191699 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:57.662609 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:57.662575 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:58.077205 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:58.077167 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:58.801130 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:58.801093 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:59.449801 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:31:59.449766 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:01.105942 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:01.105849 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:01.245834 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:01.245801 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:07.955392 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:07.955357 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:08.176010 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:08.175937 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:13.165065 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:13.165030 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:13.268154 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:13.268100 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:13.553003 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:13.552969 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:13.650586 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:13.650554 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:16.960368 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:16.960335 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:17.097842 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:17.097814 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:18.315138 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:18.315101 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:18.433071 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:18.433038 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:19.825459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:19.825423 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:20.082271 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:20.082185 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:30.739043 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.739007 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c57c47bcd-qsfqd"] Jul 17 17:32:30.744520 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.744496 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.761193 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.761168 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c57c47bcd-qsfqd"] Jul 17 17:32:30.763941 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.763905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/218a9863-42c8-4c5a-a019-225d6ade26fa-service-ca\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.764061 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.763950 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/218a9863-42c8-4c5a-a019-225d6ade26fa-console-oauth-config\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.764061 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.763998 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/218a9863-42c8-4c5a-a019-225d6ade26fa-console-config\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.764061 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.764027 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fwvf\" (UniqueName: \"kubernetes.io/projected/218a9863-42c8-4c5a-a019-225d6ade26fa-kube-api-access-6fwvf\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.764216 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.764117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/218a9863-42c8-4c5a-a019-225d6ade26fa-oauth-serving-cert\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.764216 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.764172 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/218a9863-42c8-4c5a-a019-225d6ade26fa-console-serving-cert\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.764216 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.764203 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/218a9863-42c8-4c5a-a019-225d6ade26fa-trusted-ca-bundle\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.865059 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.865027 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/218a9863-42c8-4c5a-a019-225d6ade26fa-console-serving-cert\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.865059 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.865060 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/218a9863-42c8-4c5a-a019-225d6ade26fa-trusted-ca-bundle\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.865299 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.865088 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/218a9863-42c8-4c5a-a019-225d6ade26fa-service-ca\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.865299 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.865105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/218a9863-42c8-4c5a-a019-225d6ade26fa-console-oauth-config\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.865299 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.865125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/218a9863-42c8-4c5a-a019-225d6ade26fa-console-config\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.865299 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.865274 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fwvf\" (UniqueName: \"kubernetes.io/projected/218a9863-42c8-4c5a-a019-225d6ade26fa-kube-api-access-6fwvf\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.865509 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.865383 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/218a9863-42c8-4c5a-a019-225d6ade26fa-oauth-serving-cert\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.865952 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.865925 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/218a9863-42c8-4c5a-a019-225d6ade26fa-service-ca\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.866059 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.866045 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/218a9863-42c8-4c5a-a019-225d6ade26fa-trusted-ca-bundle\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.866059 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.866008 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/218a9863-42c8-4c5a-a019-225d6ade26fa-oauth-serving-cert\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.866151 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.865960 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/218a9863-42c8-4c5a-a019-225d6ade26fa-console-config\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.867469 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.867443 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/218a9863-42c8-4c5a-a019-225d6ade26fa-console-serving-cert\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.867730 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.867530 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/218a9863-42c8-4c5a-a019-225d6ade26fa-console-oauth-config\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.891210 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.891180 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fwvf\" (UniqueName: \"kubernetes.io/projected/218a9863-42c8-4c5a-a019-225d6ade26fa-kube-api-access-6fwvf\") pod \"console-5c57c47bcd-qsfqd\" (UID: \"218a9863-42c8-4c5a-a019-225d6ade26fa\") " pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:30.908321 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:30.908300 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:31.044056 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:31.043968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:31.053986 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:31.053963 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:31.194886 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:31.194861 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c57c47bcd-qsfqd"] Jul 17 17:32:31.196289 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:32:31.196266 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218a9863_42c8_4c5a_a019_225d6ade26fa.slice/crio-d7100ab03f31ed350aeca9b7fc7cabb26de828e44c76e9df3279f5d4d3a4e913 WatchSource:0}: Error finding container d7100ab03f31ed350aeca9b7fc7cabb26de828e44c76e9df3279f5d4d3a4e913: Status 404 returned error can't find the container with id d7100ab03f31ed350aeca9b7fc7cabb26de828e44c76e9df3279f5d4d3a4e913 Jul 17 17:32:32.091272 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:32.091236 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c57c47bcd-qsfqd" event={"ID":"218a9863-42c8-4c5a-a019-225d6ade26fa","Type":"ContainerStarted","Data":"9a9927993bfea2d619895e21b465723832ac5d2ed75dd905da75a58f368263ae"} Jul 17 17:32:32.091649 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:32.091284 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c57c47bcd-qsfqd" event={"ID":"218a9863-42c8-4c5a-a019-225d6ade26fa","Type":"ContainerStarted","Data":"d7100ab03f31ed350aeca9b7fc7cabb26de828e44c76e9df3279f5d4d3a4e913"} Jul 17 17:32:41.054142 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:41.054109 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:41.054611 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:41.054189 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:41.059171 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:41.059141 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:41.085713 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:41.085671 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c57c47bcd-qsfqd" podStartSLOduration=11.085657732 podStartE2EDuration="11.085657732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:32.118728951 +0000 UTC m=+472.854153011" watchObservedRunningTime="2026-07-17 17:32:41.085657732 +0000 UTC m=+481.821081780" Jul 17 17:32:41.142729 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:41.142705 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c57c47bcd-qsfqd" Jul 17 17:32:41.206207 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:41.206177 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7697fff5c8-6fzct"] Jul 17 17:32:41.323693 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:41.323607 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:41.442260 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:41.442227 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:41.530997 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:41.530971 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:42.528434 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.528401 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk"] Jul 17 17:32:42.531982 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.531958 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.537559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.537536 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-z7vz8\"" Jul 17 17:32:42.554307 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.554283 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk"] Jul 17 17:32:42.567829 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.567803 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00110376-3ab0-40d6-915f-92ac67509c1f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.567922 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.567839 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00110376-3ab0-40d6-915f-92ac67509c1f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.567922 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.567868 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00110376-3ab0-40d6-915f-92ac67509c1f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.567922 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.567887 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00110376-3ab0-40d6-915f-92ac67509c1f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.567922 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.567904 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00110376-3ab0-40d6-915f-92ac67509c1f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.568057 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.567930 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00110376-3ab0-40d6-915f-92ac67509c1f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.568057 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.568032 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v2db\" (UniqueName: \"kubernetes.io/projected/00110376-3ab0-40d6-915f-92ac67509c1f-kube-api-access-7v2db\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.568121 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.568064 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00110376-3ab0-40d6-915f-92ac67509c1f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.568153 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.568136 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00110376-3ab0-40d6-915f-92ac67509c1f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.579991 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.579964 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:42.669560 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.669522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00110376-3ab0-40d6-915f-92ac67509c1f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.669560 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.669564 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00110376-3ab0-40d6-915f-92ac67509c1f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.669766 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.669590 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00110376-3ab0-40d6-915f-92ac67509c1f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.669766 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.669726 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00110376-3ab0-40d6-915f-92ac67509c1f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.669766 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.669759 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00110376-3ab0-40d6-915f-92ac67509c1f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.669891 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.669801 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00110376-3ab0-40d6-915f-92ac67509c1f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.669942 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.669885 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v2db\" (UniqueName: \"kubernetes.io/projected/00110376-3ab0-40d6-915f-92ac67509c1f-kube-api-access-7v2db\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.669942 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.669927 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00110376-3ab0-40d6-915f-92ac67509c1f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.670041 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.669965 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00110376-3ab0-40d6-915f-92ac67509c1f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.670041 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.669973 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00110376-3ab0-40d6-915f-92ac67509c1f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.670041 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.670009 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00110376-3ab0-40d6-915f-92ac67509c1f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.670196 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.670094 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00110376-3ab0-40d6-915f-92ac67509c1f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.670252 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.670231 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00110376-3ab0-40d6-915f-92ac67509c1f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.670467 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.670441 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00110376-3ab0-40d6-915f-92ac67509c1f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.672148 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.672128 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00110376-3ab0-40d6-915f-92ac67509c1f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.672370 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.672349 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00110376-3ab0-40d6-915f-92ac67509c1f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.679188 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.679168 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00110376-3ab0-40d6-915f-92ac67509c1f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.679395 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.679373 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v2db\" (UniqueName: \"kubernetes.io/projected/00110376-3ab0-40d6-915f-92ac67509c1f-kube-api-access-7v2db\") pod \"maas-default-gateway-openshift-default-56654dcf7-x5wdk\" (UID: \"00110376-3ab0-40d6-915f-92ac67509c1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.808953 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.808877 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:42.841905 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.841877 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:42.992013 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.991979 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk"] Jul 17 17:32:42.994764 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:32:42.994739 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00110376_3ab0_40d6_915f_92ac67509c1f.slice/crio-f7132fb3aba4d914759f941797ea5fb6138c02ca67b704b68ff291a62730315e WatchSource:0}: Error finding container f7132fb3aba4d914759f941797ea5fb6138c02ca67b704b68ff291a62730315e: Status 404 returned error can't find the container with id f7132fb3aba4d914759f941797ea5fb6138c02ca67b704b68ff291a62730315e Jul 17 17:32:42.996967 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.996936 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:32:42.997028 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.997006 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:32:42.997062 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:42.997037 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:32:43.151165 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:43.151083 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" event={"ID":"00110376-3ab0-40d6-915f-92ac67509c1f","Type":"ContainerStarted","Data":"84b0a225bbed169491c4ada9b2fc9edaa674da1d64d9a84fe9732d30cff92d64"} Jul 17 17:32:43.151165 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:43.151133 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" event={"ID":"00110376-3ab0-40d6-915f-92ac67509c1f","Type":"ContainerStarted","Data":"f7132fb3aba4d914759f941797ea5fb6138c02ca67b704b68ff291a62730315e"} Jul 17 17:32:43.180090 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:43.180029 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" podStartSLOduration=1.180007128 podStartE2EDuration="1.180007128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:43.179554943 +0000 UTC m=+483.914979012" watchObservedRunningTime="2026-07-17 17:32:43.180007128 +0000 UTC m=+483.915431179" Jul 17 17:32:43.842724 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:43.842694 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:44.450656 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:44.450608 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:44.603641 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:44.603587 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:44.717506 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:44.717428 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:44.847325 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:44.847299 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:45.160114 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:45.160081 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:45.160946 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:45.160926 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:32:45.353956 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:45.353926 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-8j4lh"] Jul 17 17:32:45.357406 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:45.357391 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8j4lh" Jul 17 17:32:45.363640 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:45.363605 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 17:32:45.364110 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:45.363871 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 17:32:45.364110 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:45.363934 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-766l6\"" Jul 17 17:32:45.372342 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:45.372321 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8j4lh"] Jul 17 17:32:45.395857 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:45.395835 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8929v\" (UniqueName: \"kubernetes.io/projected/6feca9a0-0c14-4b81-bfd4-230d972edd08-kube-api-access-8929v\") pod \"authorino-58cd955675-8j4lh\" (UID: \"6feca9a0-0c14-4b81-bfd4-230d972edd08\") " pod="kuadrant-system/authorino-58cd955675-8j4lh" Jul 17 17:32:45.424595 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:45.424522 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:45.496641 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:45.496594 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8929v\" (UniqueName: \"kubernetes.io/projected/6feca9a0-0c14-4b81-bfd4-230d972edd08-kube-api-access-8929v\") pod \"authorino-58cd955675-8j4lh\" (UID: \"6feca9a0-0c14-4b81-bfd4-230d972edd08\") " pod="kuadrant-system/authorino-58cd955675-8j4lh" Jul 17 17:32:45.508672 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:45.508644 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8929v\" (UniqueName: \"kubernetes.io/projected/6feca9a0-0c14-4b81-bfd4-230d972edd08-kube-api-access-8929v\") pod \"authorino-58cd955675-8j4lh\" (UID: \"6feca9a0-0c14-4b81-bfd4-230d972edd08\") " pod="kuadrant-system/authorino-58cd955675-8j4lh" Jul 17 17:32:45.539022 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:45.538992 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:45.666046 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:45.666015 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8j4lh" Jul 17 17:32:45.795237 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:45.795212 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8j4lh"] Jul 17 17:32:45.796661 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:32:45.796614 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6feca9a0_0c14_4b81_bfd4_230d972edd08.slice/crio-553e9f8bfef8bf2b543412f000c72291d500f76593ca7ebe6dbbebcf4333bc9f WatchSource:0}: Error finding container 553e9f8bfef8bf2b543412f000c72291d500f76593ca7ebe6dbbebcf4333bc9f: Status 404 returned error can't find the container with id 553e9f8bfef8bf2b543412f000c72291d500f76593ca7ebe6dbbebcf4333bc9f Jul 17 17:32:46.167861 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:46.167831 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8j4lh" event={"ID":"6feca9a0-0c14-4b81-bfd4-230d972edd08","Type":"ContainerStarted","Data":"553e9f8bfef8bf2b543412f000c72291d500f76593ca7ebe6dbbebcf4333bc9f"} Jul 17 17:32:49.184145 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:49.184100 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8j4lh" event={"ID":"6feca9a0-0c14-4b81-bfd4-230d972edd08","Type":"ContainerStarted","Data":"aee1a291432a50bb5d0991698329ab95ef78003435ae8fd903fc28e1c3fde9b2"} Jul 17 17:32:49.207377 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:49.207329 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-8j4lh" podStartSLOduration=0.947823055 podStartE2EDuration="4.207312081s" totalImagesPullingTime="3.259489026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:49.205816217 +0000 UTC m=+489.941240269" watchObservedRunningTime="2026-07-17 17:32:49.207312081 +0000 UTC m=+489.942736160" Jul 17 17:32:51.653025 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:51.652988 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:51.768730 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:51.768701 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:52.721811 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:52.721780 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:52.990039 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:52.990009 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8j4lh"] Jul 17 17:32:52.990194 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:52.990172 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-8j4lh" podUID="6feca9a0-0c14-4b81-bfd4-230d972edd08" containerName="authorino" containerID="cri-o://aee1a291432a50bb5d0991698329ab95ef78003435ae8fd903fc28e1c3fde9b2" gracePeriod=30 Jul 17 17:32:53.204799 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:53.204763 2573 generic.go:356] "Generic (PLEG): container finished" podID="6feca9a0-0c14-4b81-bfd4-230d972edd08" containerID="aee1a291432a50bb5d0991698329ab95ef78003435ae8fd903fc28e1c3fde9b2" exitCode=0 Jul 17 17:32:53.204959 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:53.204838 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8j4lh" event={"ID":"6feca9a0-0c14-4b81-bfd4-230d972edd08","Type":"ContainerDied","Data":"aee1a291432a50bb5d0991698329ab95ef78003435ae8fd903fc28e1c3fde9b2"} Jul 17 17:32:53.228604 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:53.228574 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8j4lh" Jul 17 17:32:53.373780 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:53.373698 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8929v\" (UniqueName: \"kubernetes.io/projected/6feca9a0-0c14-4b81-bfd4-230d972edd08-kube-api-access-8929v\") pod \"6feca9a0-0c14-4b81-bfd4-230d972edd08\" (UID: \"6feca9a0-0c14-4b81-bfd4-230d972edd08\") " Jul 17 17:32:53.375790 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:53.375767 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6feca9a0-0c14-4b81-bfd4-230d972edd08-kube-api-access-8929v" (OuterVolumeSpecName: "kube-api-access-8929v") pod "6feca9a0-0c14-4b81-bfd4-230d972edd08" (UID: "6feca9a0-0c14-4b81-bfd4-230d972edd08"). InnerVolumeSpecName "kube-api-access-8929v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:32:53.475332 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:53.475303 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8929v\" (UniqueName: \"kubernetes.io/projected/6feca9a0-0c14-4b81-bfd4-230d972edd08-kube-api-access-8929v\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:32:54.210901 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:54.210827 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8j4lh" Jul 17 17:32:54.211283 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:54.210824 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8j4lh" event={"ID":"6feca9a0-0c14-4b81-bfd4-230d972edd08","Type":"ContainerDied","Data":"553e9f8bfef8bf2b543412f000c72291d500f76593ca7ebe6dbbebcf4333bc9f"} Jul 17 17:32:54.211283 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:54.210940 2573 scope.go:117] "RemoveContainer" containerID="aee1a291432a50bb5d0991698329ab95ef78003435ae8fd903fc28e1c3fde9b2" Jul 17 17:32:54.233861 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:54.233838 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8j4lh"] Jul 17 17:32:54.238772 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:54.238752 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-8j4lh"] Jul 17 17:32:55.776752 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:32:55.776721 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6feca9a0-0c14-4b81-bfd4-230d972edd08" path="/var/lib/kubelet/pods/6feca9a0-0c14-4b81-bfd4-230d972edd08/volumes" Jul 17 17:33:05.480405 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:05.480368 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:05.577360 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:05.577329 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:05.975090 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:05.975060 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vltgs"] Jul 17 17:33:05.975459 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:05.975447 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6feca9a0-0c14-4b81-bfd4-230d972edd08" containerName="authorino" Jul 17 17:33:05.975506 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:05.975461 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6feca9a0-0c14-4b81-bfd4-230d972edd08" containerName="authorino" Jul 17 17:33:05.975539 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:05.975522 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6feca9a0-0c14-4b81-bfd4-230d972edd08" containerName="authorino" Jul 17 17:33:05.978036 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:05.978021 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vltgs" Jul 17 17:33:05.984246 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:05.984221 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 17:33:05.984402 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:05.984224 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 17 17:33:05.984895 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:05.984819 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 17:33:05.985382 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:05.985368 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-68njn\"" Jul 17 17:33:05.991451 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:05.991432 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vltgs"] Jul 17 17:33:06.047925 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.047893 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:06.089302 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.089255 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlmq6\" (UniqueName: \"kubernetes.io/projected/3cda7224-c118-4808-8117-64c508e4f92a-kube-api-access-rlmq6\") pod \"postgres-69fb4594cf-vltgs\" (UID: \"3cda7224-c118-4808-8117-64c508e4f92a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vltgs" Jul 17 17:33:06.089501 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.089378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3cda7224-c118-4808-8117-64c508e4f92a-data\") pod \"postgres-69fb4594cf-vltgs\" (UID: \"3cda7224-c118-4808-8117-64c508e4f92a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vltgs" Jul 17 17:33:06.190723 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.190687 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmq6\" (UniqueName: \"kubernetes.io/projected/3cda7224-c118-4808-8117-64c508e4f92a-kube-api-access-rlmq6\") pod \"postgres-69fb4594cf-vltgs\" (UID: \"3cda7224-c118-4808-8117-64c508e4f92a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vltgs" Jul 17 17:33:06.190903 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.190763 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3cda7224-c118-4808-8117-64c508e4f92a-data\") pod \"postgres-69fb4594cf-vltgs\" (UID: \"3cda7224-c118-4808-8117-64c508e4f92a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vltgs" Jul 17 17:33:06.191142 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.191121 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3cda7224-c118-4808-8117-64c508e4f92a-data\") pod \"postgres-69fb4594cf-vltgs\" (UID: \"3cda7224-c118-4808-8117-64c508e4f92a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vltgs" Jul 17 17:33:06.202880 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.202849 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlmq6\" (UniqueName: \"kubernetes.io/projected/3cda7224-c118-4808-8117-64c508e4f92a-kube-api-access-rlmq6\") pod \"postgres-69fb4594cf-vltgs\" (UID: \"3cda7224-c118-4808-8117-64c508e4f92a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vltgs" Jul 17 17:33:06.226443 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.226375 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7697fff5c8-6fzct" podUID="27d51da0-37a5-419c-b605-cff572bbbbed" containerName="console" containerID="cri-o://3c14a406e3aa9c3d2b3448b421ab7890a7e4728820421500db9ec61264a051d8" gracePeriod=15 Jul 17 17:33:06.288321 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.288290 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vltgs" Jul 17 17:33:06.422981 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.422949 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vltgs"] Jul 17 17:33:06.426571 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:33:06.426537 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cda7224_c118_4808_8117_64c508e4f92a.slice/crio-1460ce30a966c2c60334ddf086d7777021cca963e8bd1d12192a1287ed2bc19e WatchSource:0}: Error finding container 1460ce30a966c2c60334ddf086d7777021cca963e8bd1d12192a1287ed2bc19e: Status 404 returned error can't find the container with id 1460ce30a966c2c60334ddf086d7777021cca963e8bd1d12192a1287ed2bc19e Jul 17 17:33:06.471261 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.471239 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:33:06.594704 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.594670 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-console-config\") pod \"27d51da0-37a5-419c-b605-cff572bbbbed\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " Jul 17 17:33:06.595165 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.594720 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27d51da0-37a5-419c-b605-cff572bbbbed-console-oauth-config\") pod \"27d51da0-37a5-419c-b605-cff572bbbbed\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " Jul 17 17:33:06.595165 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.594768 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mf6fd\" (UniqueName: \"kubernetes.io/projected/27d51da0-37a5-419c-b605-cff572bbbbed-kube-api-access-mf6fd\") pod \"27d51da0-37a5-419c-b605-cff572bbbbed\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " Jul 17 17:33:06.595165 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.594819 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-service-ca\") pod \"27d51da0-37a5-419c-b605-cff572bbbbed\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " Jul 17 17:33:06.595165 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.594848 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-trusted-ca-bundle\") pod \"27d51da0-37a5-419c-b605-cff572bbbbed\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " Jul 17 17:33:06.595165 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.594880 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-oauth-serving-cert\") pod \"27d51da0-37a5-419c-b605-cff572bbbbed\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " Jul 17 17:33:06.595165 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.595002 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27d51da0-37a5-419c-b605-cff572bbbbed-console-serving-cert\") pod \"27d51da0-37a5-419c-b605-cff572bbbbed\" (UID: \"27d51da0-37a5-419c-b605-cff572bbbbed\") " Jul 17 17:33:06.595165 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.595072 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-console-config" (OuterVolumeSpecName: "console-config") pod "27d51da0-37a5-419c-b605-cff572bbbbed" (UID: "27d51da0-37a5-419c-b605-cff572bbbbed"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:33:06.595515 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.595277 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-service-ca" (OuterVolumeSpecName: "service-ca") pod "27d51da0-37a5-419c-b605-cff572bbbbed" (UID: "27d51da0-37a5-419c-b605-cff572bbbbed"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:33:06.595515 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.595343 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "27d51da0-37a5-419c-b605-cff572bbbbed" (UID: "27d51da0-37a5-419c-b605-cff572bbbbed"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:33:06.595515 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.595470 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:33:06.595515 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.595490 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-console-config\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:33:06.595785 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.595577 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "27d51da0-37a5-419c-b605-cff572bbbbed" (UID: "27d51da0-37a5-419c-b605-cff572bbbbed"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:33:06.597022 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.596994 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d51da0-37a5-419c-b605-cff572bbbbed-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "27d51da0-37a5-419c-b605-cff572bbbbed" (UID: "27d51da0-37a5-419c-b605-cff572bbbbed"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:06.597113 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.597023 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d51da0-37a5-419c-b605-cff572bbbbed-kube-api-access-mf6fd" (OuterVolumeSpecName: "kube-api-access-mf6fd") pod "27d51da0-37a5-419c-b605-cff572bbbbed" (UID: "27d51da0-37a5-419c-b605-cff572bbbbed"). InnerVolumeSpecName "kube-api-access-mf6fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:33:06.597113 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.597046 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d51da0-37a5-419c-b605-cff572bbbbed-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "27d51da0-37a5-419c-b605-cff572bbbbed" (UID: "27d51da0-37a5-419c-b605-cff572bbbbed"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:06.696958 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.696925 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27d51da0-37a5-419c-b605-cff572bbbbed-console-serving-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:33:06.696958 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.696951 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27d51da0-37a5-419c-b605-cff572bbbbed-console-oauth-config\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:33:06.696958 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.696961 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf6fd\" (UniqueName: \"kubernetes.io/projected/27d51da0-37a5-419c-b605-cff572bbbbed-kube-api-access-mf6fd\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:33:06.697169 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.696972 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-trusted-ca-bundle\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:33:06.697169 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:06.696981 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27d51da0-37a5-419c-b605-cff572bbbbed-oauth-serving-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:33:07.272690 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:07.272654 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vltgs" event={"ID":"3cda7224-c118-4808-8117-64c508e4f92a","Type":"ContainerStarted","Data":"1460ce30a966c2c60334ddf086d7777021cca963e8bd1d12192a1287ed2bc19e"} Jul 17 17:33:07.273938 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:07.273914 2573 generic.go:356] "Generic (PLEG): container finished" podID="27d51da0-37a5-419c-b605-cff572bbbbed" containerID="3c14a406e3aa9c3d2b3448b421ab7890a7e4728820421500db9ec61264a051d8" exitCode=0 Jul 17 17:33:07.274066 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:07.273955 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7697fff5c8-6fzct" event={"ID":"27d51da0-37a5-419c-b605-cff572bbbbed","Type":"ContainerDied","Data":"3c14a406e3aa9c3d2b3448b421ab7890a7e4728820421500db9ec61264a051d8"} Jul 17 17:33:07.274066 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:07.273981 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7697fff5c8-6fzct" event={"ID":"27d51da0-37a5-419c-b605-cff572bbbbed","Type":"ContainerDied","Data":"485349c24902fc4a8084d4c42b0050cb011745e42285a83339fcc866d93390cc"} Jul 17 17:33:07.274066 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:07.273988 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7697fff5c8-6fzct" Jul 17 17:33:07.274066 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:07.273999 2573 scope.go:117] "RemoveContainer" containerID="3c14a406e3aa9c3d2b3448b421ab7890a7e4728820421500db9ec61264a051d8" Jul 17 17:33:07.282425 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:07.282405 2573 scope.go:117] "RemoveContainer" containerID="3c14a406e3aa9c3d2b3448b421ab7890a7e4728820421500db9ec61264a051d8" Jul 17 17:33:07.282703 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:33:07.282683 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c14a406e3aa9c3d2b3448b421ab7890a7e4728820421500db9ec61264a051d8\": container with ID starting with 3c14a406e3aa9c3d2b3448b421ab7890a7e4728820421500db9ec61264a051d8 not found: ID does not exist" containerID="3c14a406e3aa9c3d2b3448b421ab7890a7e4728820421500db9ec61264a051d8" Jul 17 17:33:07.282764 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:07.282714 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c14a406e3aa9c3d2b3448b421ab7890a7e4728820421500db9ec61264a051d8"} err="failed to get container status \"3c14a406e3aa9c3d2b3448b421ab7890a7e4728820421500db9ec61264a051d8\": rpc error: code = NotFound desc = could not find container \"3c14a406e3aa9c3d2b3448b421ab7890a7e4728820421500db9ec61264a051d8\": container with ID starting with 3c14a406e3aa9c3d2b3448b421ab7890a7e4728820421500db9ec61264a051d8 not found: ID does not exist" Jul 17 17:33:07.296425 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:07.296405 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7697fff5c8-6fzct"] Jul 17 17:33:07.300403 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:07.300384 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7697fff5c8-6fzct"] Jul 17 17:33:07.777554 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:07.777519 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27d51da0-37a5-419c-b605-cff572bbbbed" path="/var/lib/kubelet/pods/27d51da0-37a5-419c-b605-cff572bbbbed/volumes" Jul 17 17:33:12.307346 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:12.307309 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vltgs" event={"ID":"3cda7224-c118-4808-8117-64c508e4f92a","Type":"ContainerStarted","Data":"0fa13bba6ce8e659c58804f4a8afd48d139fad9dc2a572a7da411e65b048bbe0"} Jul 17 17:33:12.307775 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:12.307507 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vltgs" Jul 17 17:33:12.348973 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:12.348925 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vltgs" podStartSLOduration=1.637535923 podStartE2EDuration="7.348911129s" totalImagesPullingTime="5.711375206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:33:12.344246904 +0000 UTC m=+513.079670972" watchObservedRunningTime="2026-07-17 17:33:12.348911129 +0000 UTC m=+513.084335177" Jul 17 17:33:18.351536 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:18.351505 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vltgs" Jul 17 17:33:18.737057 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:18.737020 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:18.892152 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:18.892120 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:26.226795 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:26.226761 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:26.327127 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:26.327088 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86f88957c8-dx6v8"] Jul 17 17:33:26.327555 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:26.327536 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d51da0-37a5-419c-b605-cff572bbbbed" containerName="console" Jul 17 17:33:26.327685 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:26.327557 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d51da0-37a5-419c-b605-cff572bbbbed" containerName="console" Jul 17 17:33:26.327685 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:26.327658 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="27d51da0-37a5-419c-b605-cff572bbbbed" containerName="console" Jul 17 17:33:26.330879 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:26.330857 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86f88957c8-dx6v8" Jul 17 17:33:26.337179 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:33:26.337141 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-controller-86f88957c8-dx6v8\" is forbidden: User \"system:node:ip-10-0-132-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-183.ec2.internal' and this object" podUID="8a066fbe-f124-4c80-8cd4-900995271d1c" pod="opendatahub/maas-controller-86f88957c8-dx6v8" Jul 17 17:33:26.337314 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:33:26.337286 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"maas-parameters\" is forbidden: User \"system:node:ip-10-0-132-183.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-183.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"maas-parameters\"" type="*v1.ConfigMap" Jul 17 17:33:26.337693 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:33:26.337381 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-controller-dockercfg-vx4rq\" is forbidden: User \"system:node:ip-10-0-132-183.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-183.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vx4rq\"" type="*v1.Secret" Jul 17 17:33:26.337793 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:33:26.337767 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-132-183.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-183.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" type="*v1.Secret" Jul 17 17:33:26.384853 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:26.384820 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbvnr\" (UniqueName: \"kubernetes.io/projected/8a066fbe-f124-4c80-8cd4-900995271d1c-kube-api-access-rbvnr\") pod \"maas-controller-86f88957c8-dx6v8\" (UID: \"8a066fbe-f124-4c80-8cd4-900995271d1c\") " pod="opendatahub/maas-controller-86f88957c8-dx6v8" Jul 17 17:33:26.385002 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:26.384878 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a066fbe-f124-4c80-8cd4-900995271d1c-webhook-cert\") pod \"maas-controller-86f88957c8-dx6v8\" (UID: \"8a066fbe-f124-4c80-8cd4-900995271d1c\") " pod="opendatahub/maas-controller-86f88957c8-dx6v8" Jul 17 17:33:26.397009 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:26.396974 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86f88957c8-dx6v8"] Jul 17 17:33:26.483112 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:26.483010 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:26.485713 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:26.485685 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbvnr\" (UniqueName: \"kubernetes.io/projected/8a066fbe-f124-4c80-8cd4-900995271d1c-kube-api-access-rbvnr\") pod \"maas-controller-86f88957c8-dx6v8\" (UID: \"8a066fbe-f124-4c80-8cd4-900995271d1c\") " pod="opendatahub/maas-controller-86f88957c8-dx6v8" Jul 17 17:33:26.485825 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:26.485736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a066fbe-f124-4c80-8cd4-900995271d1c-webhook-cert\") pod \"maas-controller-86f88957c8-dx6v8\" (UID: \"8a066fbe-f124-4c80-8cd4-900995271d1c\") " pod="opendatahub/maas-controller-86f88957c8-dx6v8" Jul 17 17:33:26.499279 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:26.499254 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbvnr\" (UniqueName: \"kubernetes.io/projected/8a066fbe-f124-4c80-8cd4-900995271d1c-kube-api-access-rbvnr\") pod \"maas-controller-86f88957c8-dx6v8\" (UID: \"8a066fbe-f124-4c80-8cd4-900995271d1c\") " pod="opendatahub/maas-controller-86f88957c8-dx6v8" Jul 17 17:33:26.680769 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:26.680735 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:27.486727 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:33:27.486694 2573 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: failed to sync secret cache: timed out waiting for the condition Jul 17 17:33:27.487111 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:33:27.486770 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a066fbe-f124-4c80-8cd4-900995271d1c-webhook-cert podName:8a066fbe-f124-4c80-8cd4-900995271d1c nodeName:}" failed. No retries permitted until 2026-07-17 17:33:27.986753306 +0000 UTC m=+528.722177338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/8a066fbe-f124-4c80-8cd4-900995271d1c-webhook-cert") pod "maas-controller-86f88957c8-dx6v8" (UID: "8a066fbe-f124-4c80-8cd4-900995271d1c") : failed to sync secret cache: timed out waiting for the condition Jul 17 17:33:27.572251 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:27.572224 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 17 17:33:27.584070 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:27.584048 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 17 17:33:27.774006 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:27.773923 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vx4rq\"" Jul 17 17:33:28.002048 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:28.002016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a066fbe-f124-4c80-8cd4-900995271d1c-webhook-cert\") pod \"maas-controller-86f88957c8-dx6v8\" (UID: \"8a066fbe-f124-4c80-8cd4-900995271d1c\") " pod="opendatahub/maas-controller-86f88957c8-dx6v8" Jul 17 17:33:28.004420 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:28.004394 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a066fbe-f124-4c80-8cd4-900995271d1c-webhook-cert\") pod \"maas-controller-86f88957c8-dx6v8\" (UID: \"8a066fbe-f124-4c80-8cd4-900995271d1c\") " pod="opendatahub/maas-controller-86f88957c8-dx6v8" Jul 17 17:33:28.140673 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:28.140567 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86f88957c8-dx6v8" Jul 17 17:33:28.273940 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:28.273912 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86f88957c8-dx6v8"] Jul 17 17:33:28.276140 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:33:28.276113 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a066fbe_f124_4c80_8cd4_900995271d1c.slice/crio-1b8cefbf67adff8860f9cd4511f8dba2659aea43193e4798cec484940e814733 WatchSource:0}: Error finding container 1b8cefbf67adff8860f9cd4511f8dba2659aea43193e4798cec484940e814733: Status 404 returned error can't find the container with id 1b8cefbf67adff8860f9cd4511f8dba2659aea43193e4798cec484940e814733 Jul 17 17:33:28.399341 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:28.399257 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86f88957c8-dx6v8" event={"ID":"8a066fbe-f124-4c80-8cd4-900995271d1c","Type":"ContainerStarted","Data":"1b8cefbf67adff8860f9cd4511f8dba2659aea43193e4798cec484940e814733"} Jul 17 17:33:29.609336 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:29.609303 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:29.739788 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:29.739757 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:31.416682 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:31.416646 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86f88957c8-dx6v8" event={"ID":"8a066fbe-f124-4c80-8cd4-900995271d1c","Type":"ContainerStarted","Data":"6824d01f116ec12a157d306b13f8365da84e43ceffc6007541afda7b4b123f2b"} Jul 17 17:33:31.417138 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:31.416779 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86f88957c8-dx6v8" Jul 17 17:33:31.445265 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:31.445207 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86f88957c8-dx6v8" podStartSLOduration=2.937971547 podStartE2EDuration="5.445192079s" totalImagesPullingTime="2.507220532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:33:31.444558574 +0000 UTC m=+532.179982623" watchObservedRunningTime="2026-07-17 17:33:31.445192079 +0000 UTC m=+532.180616128" Jul 17 17:33:33.840165 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:33.840128 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:33.919387 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:33.919353 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:41.204134 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.204099 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86f88957c8-dx6v8"] Jul 17 17:33:41.204596 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.204303 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-86f88957c8-dx6v8" podUID="8a066fbe-f124-4c80-8cd4-900995271d1c" containerName="manager" containerID="cri-o://6824d01f116ec12a157d306b13f8365da84e43ceffc6007541afda7b4b123f2b" gracePeriod=30 Jul 17 17:33:41.206735 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.206712 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86f88957c8-dx6v8" Jul 17 17:33:41.236204 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:33:41.236168 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a066fbe_f124_4c80_8cd4_900995271d1c.slice/crio-6824d01f116ec12a157d306b13f8365da84e43ceffc6007541afda7b4b123f2b.scope\": RecentStats: unable to find data in memory cache]" Jul 17 17:33:41.448042 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.448019 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86f88957c8-dx6v8" Jul 17 17:33:41.466817 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.466750 2573 generic.go:356] "Generic (PLEG): container finished" podID="8a066fbe-f124-4c80-8cd4-900995271d1c" containerID="6824d01f116ec12a157d306b13f8365da84e43ceffc6007541afda7b4b123f2b" exitCode=0 Jul 17 17:33:41.466817 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.466793 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86f88957c8-dx6v8" Jul 17 17:33:41.466965 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.466836 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86f88957c8-dx6v8" event={"ID":"8a066fbe-f124-4c80-8cd4-900995271d1c","Type":"ContainerDied","Data":"6824d01f116ec12a157d306b13f8365da84e43ceffc6007541afda7b4b123f2b"} Jul 17 17:33:41.466965 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.466913 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86f88957c8-dx6v8" event={"ID":"8a066fbe-f124-4c80-8cd4-900995271d1c","Type":"ContainerDied","Data":"1b8cefbf67adff8860f9cd4511f8dba2659aea43193e4798cec484940e814733"} Jul 17 17:33:41.466965 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.466935 2573 scope.go:117] "RemoveContainer" containerID="6824d01f116ec12a157d306b13f8365da84e43ceffc6007541afda7b4b123f2b" Jul 17 17:33:41.476684 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.476519 2573 scope.go:117] "RemoveContainer" containerID="6824d01f116ec12a157d306b13f8365da84e43ceffc6007541afda7b4b123f2b" Jul 17 17:33:41.476847 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:33:41.476827 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6824d01f116ec12a157d306b13f8365da84e43ceffc6007541afda7b4b123f2b\": container with ID starting with 6824d01f116ec12a157d306b13f8365da84e43ceffc6007541afda7b4b123f2b not found: ID does not exist" containerID="6824d01f116ec12a157d306b13f8365da84e43ceffc6007541afda7b4b123f2b" Jul 17 17:33:41.476922 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.476855 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6824d01f116ec12a157d306b13f8365da84e43ceffc6007541afda7b4b123f2b"} err="failed to get container status \"6824d01f116ec12a157d306b13f8365da84e43ceffc6007541afda7b4b123f2b\": rpc error: code = NotFound desc = could not find container \"6824d01f116ec12a157d306b13f8365da84e43ceffc6007541afda7b4b123f2b\": container with ID starting with 6824d01f116ec12a157d306b13f8365da84e43ceffc6007541afda7b4b123f2b not found: ID does not exist" Jul 17 17:33:41.528760 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.528729 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbvnr\" (UniqueName: \"kubernetes.io/projected/8a066fbe-f124-4c80-8cd4-900995271d1c-kube-api-access-rbvnr\") pod \"8a066fbe-f124-4c80-8cd4-900995271d1c\" (UID: \"8a066fbe-f124-4c80-8cd4-900995271d1c\") " Jul 17 17:33:41.528905 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.528773 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a066fbe-f124-4c80-8cd4-900995271d1c-webhook-cert\") pod \"8a066fbe-f124-4c80-8cd4-900995271d1c\" (UID: \"8a066fbe-f124-4c80-8cd4-900995271d1c\") " Jul 17 17:33:41.530935 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.530910 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a066fbe-f124-4c80-8cd4-900995271d1c-kube-api-access-rbvnr" (OuterVolumeSpecName: "kube-api-access-rbvnr") pod "8a066fbe-f124-4c80-8cd4-900995271d1c" (UID: "8a066fbe-f124-4c80-8cd4-900995271d1c"). InnerVolumeSpecName "kube-api-access-rbvnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:33:41.531022 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.530956 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a066fbe-f124-4c80-8cd4-900995271d1c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8a066fbe-f124-4c80-8cd4-900995271d1c" (UID: "8a066fbe-f124-4c80-8cd4-900995271d1c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:41.630078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.630042 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbvnr\" (UniqueName: \"kubernetes.io/projected/8a066fbe-f124-4c80-8cd4-900995271d1c-kube-api-access-rbvnr\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:33:41.630078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.630072 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a066fbe-f124-4c80-8cd4-900995271d1c-webhook-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:33:41.795306 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.795273 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86f88957c8-dx6v8"] Jul 17 17:33:41.800785 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:41.800761 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86f88957c8-dx6v8"] Jul 17 17:33:43.777294 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:33:43.777262 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a066fbe-f124-4c80-8cd4-900995271d1c" path="/var/lib/kubelet/pods/8a066fbe-f124-4c80-8cd4-900995271d1c/volumes" Jul 17 17:34:01.390684 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:01.390653 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:01.518343 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:01.518306 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:02.131770 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.131729 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:02.237824 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.237784 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-c756574f6-ksprx"] Jul 17 17:34:02.238261 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.238243 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a066fbe-f124-4c80-8cd4-900995271d1c" containerName="manager" Jul 17 17:34:02.238416 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.238262 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a066fbe-f124-4c80-8cd4-900995271d1c" containerName="manager" Jul 17 17:34:02.238416 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.238341 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a066fbe-f124-4c80-8cd4-900995271d1c" containerName="manager" Jul 17 17:34:02.258381 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.258355 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c756574f6-ksprx" Jul 17 17:34:02.263126 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.263103 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 17 17:34:02.263252 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.263170 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h76pq\"" Jul 17 17:34:02.291164 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.291140 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c756574f6-ksprx"] Jul 17 17:34:02.319094 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.319055 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s72hg\" (UniqueName: \"kubernetes.io/projected/fa2cf22b-a4ca-4849-bf72-7982d45cca1b-kube-api-access-s72hg\") pod \"maas-api-c756574f6-ksprx\" (UID: \"fa2cf22b-a4ca-4849-bf72-7982d45cca1b\") " pod="odh-ai-gateway-infra/maas-api-c756574f6-ksprx" Jul 17 17:34:02.319228 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.319107 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2cf22b-a4ca-4849-bf72-7982d45cca1b-openshift-service-ca\") pod \"maas-api-c756574f6-ksprx\" (UID: \"fa2cf22b-a4ca-4849-bf72-7982d45cca1b\") " pod="odh-ai-gateway-infra/maas-api-c756574f6-ksprx" Jul 17 17:34:02.319228 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.319176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa2cf22b-a4ca-4849-bf72-7982d45cca1b-maas-api-tls\") pod \"maas-api-c756574f6-ksprx\" (UID: \"fa2cf22b-a4ca-4849-bf72-7982d45cca1b\") " pod="odh-ai-gateway-infra/maas-api-c756574f6-ksprx" Jul 17 17:34:02.420811 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.420730 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa2cf22b-a4ca-4849-bf72-7982d45cca1b-maas-api-tls\") pod \"maas-api-c756574f6-ksprx\" (UID: \"fa2cf22b-a4ca-4849-bf72-7982d45cca1b\") " pod="odh-ai-gateway-infra/maas-api-c756574f6-ksprx" Jul 17 17:34:02.421171 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.420816 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s72hg\" (UniqueName: \"kubernetes.io/projected/fa2cf22b-a4ca-4849-bf72-7982d45cca1b-kube-api-access-s72hg\") pod \"maas-api-c756574f6-ksprx\" (UID: \"fa2cf22b-a4ca-4849-bf72-7982d45cca1b\") " pod="odh-ai-gateway-infra/maas-api-c756574f6-ksprx" Jul 17 17:34:02.421171 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.420837 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2cf22b-a4ca-4849-bf72-7982d45cca1b-openshift-service-ca\") pod \"maas-api-c756574f6-ksprx\" (UID: \"fa2cf22b-a4ca-4849-bf72-7982d45cca1b\") " pod="odh-ai-gateway-infra/maas-api-c756574f6-ksprx" Jul 17 17:34:02.421405 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.421387 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2cf22b-a4ca-4849-bf72-7982d45cca1b-openshift-service-ca\") pod \"maas-api-c756574f6-ksprx\" (UID: \"fa2cf22b-a4ca-4849-bf72-7982d45cca1b\") " pod="odh-ai-gateway-infra/maas-api-c756574f6-ksprx" Jul 17 17:34:02.423160 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.423141 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa2cf22b-a4ca-4849-bf72-7982d45cca1b-maas-api-tls\") pod \"maas-api-c756574f6-ksprx\" (UID: \"fa2cf22b-a4ca-4849-bf72-7982d45cca1b\") " pod="odh-ai-gateway-infra/maas-api-c756574f6-ksprx" Jul 17 17:34:02.448369 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.448347 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s72hg\" (UniqueName: \"kubernetes.io/projected/fa2cf22b-a4ca-4849-bf72-7982d45cca1b-kube-api-access-s72hg\") pod \"maas-api-c756574f6-ksprx\" (UID: \"fa2cf22b-a4ca-4849-bf72-7982d45cca1b\") " pod="odh-ai-gateway-infra/maas-api-c756574f6-ksprx" Jul 17 17:34:02.493305 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.493274 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:02.568271 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.568240 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c756574f6-ksprx" Jul 17 17:34:02.734419 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.734257 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c756574f6-ksprx"] Jul 17 17:34:02.737211 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:34:02.737182 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2cf22b_a4ca_4849_bf72_7982d45cca1b.slice/crio-e6512b0451f804bf6f0ba26189bb0a5ac70f599fc2beccc268645493b5dcc0ad WatchSource:0}: Error finding container e6512b0451f804bf6f0ba26189bb0a5ac70f599fc2beccc268645493b5dcc0ad: Status 404 returned error can't find the container with id e6512b0451f804bf6f0ba26189bb0a5ac70f599fc2beccc268645493b5dcc0ad Jul 17 17:34:02.927641 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:02.927593 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:03.059071 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:03.058981 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:03.569248 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:03.569211 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c756574f6-ksprx" event={"ID":"fa2cf22b-a4ca-4849-bf72-7982d45cca1b","Type":"ContainerStarted","Data":"e6512b0451f804bf6f0ba26189bb0a5ac70f599fc2beccc268645493b5dcc0ad"} Jul 17 17:34:05.579511 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:05.579475 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c756574f6-ksprx" event={"ID":"fa2cf22b-a4ca-4849-bf72-7982d45cca1b","Type":"ContainerStarted","Data":"57e6db6758452ed28cc5868227bc02df015996546aa5b7ee0ca9f36bd83dc956"} Jul 17 17:34:05.580015 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:05.579559 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-c756574f6-ksprx" Jul 17 17:34:05.617440 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:05.617378 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-c756574f6-ksprx" podStartSLOduration=1.211303078 podStartE2EDuration="3.6173582s" totalImagesPullingTime="2.406055122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:34:05.609789198 +0000 UTC m=+566.345213248" watchObservedRunningTime="2026-07-17 17:34:05.6173582 +0000 UTC m=+566.352782252" Jul 17 17:34:11.590844 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:11.590808 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-c756574f6-ksprx" Jul 17 17:34:11.609554 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:11.609527 2573 generic.go:356] "Generic (PLEG): container finished" podID="00110376-3ab0-40d6-915f-92ac67509c1f" containerID="84b0a225bbed169491c4ada9b2fc9edaa674da1d64d9a84fe9732d30cff92d64" exitCode=0 Jul 17 17:34:11.609718 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:11.609600 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" event={"ID":"00110376-3ab0-40d6-915f-92ac67509c1f","Type":"ContainerDied","Data":"84b0a225bbed169491c4ada9b2fc9edaa674da1d64d9a84fe9732d30cff92d64"} Jul 17 17:34:11.610031 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:11.610017 2573 scope.go:117] "RemoveContainer" containerID="84b0a225bbed169491c4ada9b2fc9edaa674da1d64d9a84fe9732d30cff92d64" Jul 17 17:34:11.611085 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:11.611055 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:34:11.611187 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:11.611149 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:34:11.611247 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:11.611207 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:34:12.630689 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:12.630653 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" event={"ID":"00110376-3ab0-40d6-915f-92ac67509c1f","Type":"ContainerStarted","Data":"e4fd625a0b5b3f1e6f96dc97c8f94a35c78a72329ee3cf0d2a0f1c92708aaa28"} Jul 17 17:34:12.842224 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:12.842182 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:34:12.843017 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:12.842993 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-x5wdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 17 17:34:12.843071 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:12.843043 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" podUID="00110376-3ab0-40d6-915f-92ac67509c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 17 17:34:13.183282 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:13.183253 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:13.283980 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:13.283948 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:13.842710 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:13.842674 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-x5wdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 17 17:34:13.843110 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:13.842733 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" podUID="00110376-3ab0-40d6-915f-92ac67509c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 17 17:34:14.843189 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:14.843150 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-x5wdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 17 17:34:14.843672 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:14.843217 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" podUID="00110376-3ab0-40d6-915f-92ac67509c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 17 17:34:15.160064 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:15.159974 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:34:15.842821 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:15.842788 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-x5wdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 17 17:34:15.842997 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:15.842855 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" podUID="00110376-3ab0-40d6-915f-92ac67509c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 17 17:34:16.846544 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:16.846517 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:34:16.847446 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:16.847424 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-x5wdk" Jul 17 17:34:25.413674 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:25.413590 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:39.683389 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:39.683359 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:34:39.687028 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:39.687008 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:34:49.507063 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:34:49.507032 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:00.783938 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:35:00.783841 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:08.716739 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:35:08.716699 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:10.628295 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:35:10.628265 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:16.241147 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:35:16.241110 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:32.520064 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:35:32.520032 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:54.907437 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:35:54.907397 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:55.007047 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:35:55.007012 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:36:53.179895 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:53.179856 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-dbfb57dbf-tjmnx"] Jul 17 17:36:53.182432 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:53.182416 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dbfb57dbf-tjmnx" Jul 17 17:36:53.190147 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:36:53.190118 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"authorino-authorino-dockercfg-766l6\" is forbidden: User \"system:node:ip-10-0-132-183.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-183.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-766l6\"" type="*v1.Secret" Jul 17 17:36:53.190249 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:36:53.190138 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"authorino-server-cert\" is forbidden: User \"system:node:ip-10-0-132-183.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-183.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" type="*v1.Secret" Jul 17 17:36:53.190249 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:36:53.190186 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-132-183.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-183.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 17 17:36:53.190249 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:36:53.190194 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-132-183.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-183.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 17 17:36:53.190360 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:36:53.190294 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"authorino-dbfb57dbf-tjmnx\" is forbidden: User \"system:node:ip-10-0-132-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-183.ec2.internal' and this object" podUID="439e3fae-7a31-4cba-842a-5a961fc9039b" pod="kuadrant-system/authorino-dbfb57dbf-tjmnx" Jul 17 17:36:53.211449 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:53.211423 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dbfb57dbf-tjmnx"] Jul 17 17:36:53.254843 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:53.254812 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/439e3fae-7a31-4cba-842a-5a961fc9039b-tls-cert\") pod \"authorino-dbfb57dbf-tjmnx\" (UID: \"439e3fae-7a31-4cba-842a-5a961fc9039b\") " pod="kuadrant-system/authorino-dbfb57dbf-tjmnx" Jul 17 17:36:53.255005 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:53.254885 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2qg8\" (UniqueName: \"kubernetes.io/projected/439e3fae-7a31-4cba-842a-5a961fc9039b-kube-api-access-h2qg8\") pod \"authorino-dbfb57dbf-tjmnx\" (UID: \"439e3fae-7a31-4cba-842a-5a961fc9039b\") " pod="kuadrant-system/authorino-dbfb57dbf-tjmnx" Jul 17 17:36:53.355798 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:53.355762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2qg8\" (UniqueName: \"kubernetes.io/projected/439e3fae-7a31-4cba-842a-5a961fc9039b-kube-api-access-h2qg8\") pod \"authorino-dbfb57dbf-tjmnx\" (UID: \"439e3fae-7a31-4cba-842a-5a961fc9039b\") " pod="kuadrant-system/authorino-dbfb57dbf-tjmnx" Jul 17 17:36:53.355978 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:53.355855 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/439e3fae-7a31-4cba-842a-5a961fc9039b-tls-cert\") pod \"authorino-dbfb57dbf-tjmnx\" (UID: \"439e3fae-7a31-4cba-842a-5a961fc9039b\") " pod="kuadrant-system/authorino-dbfb57dbf-tjmnx" Jul 17 17:36:53.450452 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:53.450378 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:36:53.568080 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:53.568045 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:36:54.164006 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:54.163976 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 17 17:36:54.168961 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:54.168940 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 17:36:54.169396 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:54.169375 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/439e3fae-7a31-4cba-842a-5a961fc9039b-tls-cert\") pod \"authorino-dbfb57dbf-tjmnx\" (UID: \"439e3fae-7a31-4cba-842a-5a961fc9039b\") " pod="kuadrant-system/authorino-dbfb57dbf-tjmnx" Jul 17 17:36:54.295179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:54.295142 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-766l6\"" Jul 17 17:36:54.542321 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:54.542293 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 17:36:54.544714 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:54.544688 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2qg8\" (UniqueName: \"kubernetes.io/projected/439e3fae-7a31-4cba-842a-5a961fc9039b-kube-api-access-h2qg8\") pod \"authorino-dbfb57dbf-tjmnx\" (UID: \"439e3fae-7a31-4cba-842a-5a961fc9039b\") " pod="kuadrant-system/authorino-dbfb57dbf-tjmnx" Jul 17 17:36:54.604199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:54.604162 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:36:54.692154 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:54.692122 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dbfb57dbf-tjmnx" Jul 17 17:36:54.701832 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:54.701674 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:36:54.821448 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:54.821370 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dbfb57dbf-tjmnx"] Jul 17 17:36:54.824728 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:36:54.824698 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod439e3fae_7a31_4cba_842a_5a961fc9039b.slice/crio-5357e90a3d0154527a32862beb9bbfa535b2e0c1ba135e6bef90ae54e151709f WatchSource:0}: Error finding container 5357e90a3d0154527a32862beb9bbfa535b2e0c1ba135e6bef90ae54e151709f: Status 404 returned error can't find the container with id 5357e90a3d0154527a32862beb9bbfa535b2e0c1ba135e6bef90ae54e151709f Jul 17 17:36:54.826341 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:54.826318 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 17:36:55.405703 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:55.405672 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dbfb57dbf-tjmnx" event={"ID":"439e3fae-7a31-4cba-842a-5a961fc9039b","Type":"ContainerStarted","Data":"818f4009f915f165ea6f192cb912d9f4f0d92ce8dee07c4672194ca295ada846"} Jul 17 17:36:55.405703 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:55.405712 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dbfb57dbf-tjmnx" event={"ID":"439e3fae-7a31-4cba-842a-5a961fc9039b","Type":"ContainerStarted","Data":"5357e90a3d0154527a32862beb9bbfa535b2e0c1ba135e6bef90ae54e151709f"} Jul 17 17:36:55.451578 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:36:55.451526 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-dbfb57dbf-tjmnx" podStartSLOduration=1.987063509 podStartE2EDuration="2.451510052s" totalImagesPullingTime="464.446543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:36:55.44779149 +0000 UTC m=+736.183215538" watchObservedRunningTime="2026-07-17 17:36:55.451510052 +0000 UTC m=+736.186934100" Jul 17 17:37:01.124768 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:37:01.124733 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:37.336660 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:37:37.336598 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:39.136640 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:37:39.136574 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:42.133361 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:42.133324 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-54f8d57c9d-xxmrr"] Jul 17 17:38:42.136992 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:42.136975 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" Jul 17 17:38:42.142435 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:42.142413 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 17 17:38:42.143700 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:42.143683 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vx4rq\"" Jul 17 17:38:42.143700 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:42.143690 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 17 17:38:42.148872 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:42.148849 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54f8d57c9d-xxmrr"] Jul 17 17:38:42.309525 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:42.309486 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm886\" (UniqueName: \"kubernetes.io/projected/389f5ad4-5a3e-483f-90da-4b80791d0f0c-kube-api-access-qm886\") pod \"maas-controller-54f8d57c9d-xxmrr\" (UID: \"389f5ad4-5a3e-483f-90da-4b80791d0f0c\") " pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" Jul 17 17:38:42.309727 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:42.309572 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/389f5ad4-5a3e-483f-90da-4b80791d0f0c-webhook-cert\") pod \"maas-controller-54f8d57c9d-xxmrr\" (UID: \"389f5ad4-5a3e-483f-90da-4b80791d0f0c\") " pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" Jul 17 17:38:42.410733 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:42.410655 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/389f5ad4-5a3e-483f-90da-4b80791d0f0c-webhook-cert\") pod \"maas-controller-54f8d57c9d-xxmrr\" (UID: \"389f5ad4-5a3e-483f-90da-4b80791d0f0c\") " pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" Jul 17 17:38:42.410875 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:42.410741 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm886\" (UniqueName: \"kubernetes.io/projected/389f5ad4-5a3e-483f-90da-4b80791d0f0c-kube-api-access-qm886\") pod \"maas-controller-54f8d57c9d-xxmrr\" (UID: \"389f5ad4-5a3e-483f-90da-4b80791d0f0c\") " pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" Jul 17 17:38:42.413256 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:42.413234 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/389f5ad4-5a3e-483f-90da-4b80791d0f0c-webhook-cert\") pod \"maas-controller-54f8d57c9d-xxmrr\" (UID: \"389f5ad4-5a3e-483f-90da-4b80791d0f0c\") " pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" Jul 17 17:38:42.421424 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:42.421394 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm886\" (UniqueName: \"kubernetes.io/projected/389f5ad4-5a3e-483f-90da-4b80791d0f0c-kube-api-access-qm886\") pod \"maas-controller-54f8d57c9d-xxmrr\" (UID: \"389f5ad4-5a3e-483f-90da-4b80791d0f0c\") " pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" Jul 17 17:38:42.447187 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:42.447164 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" Jul 17 17:38:42.577011 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:42.576854 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54f8d57c9d-xxmrr"] Jul 17 17:38:42.579244 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:38:42.579202 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389f5ad4_5a3e_483f_90da_4b80791d0f0c.slice/crio-5c52d95cacb5cd894ba84277d418343d47f0303505ef132b69676465d909f6e3 WatchSource:0}: Error finding container 5c52d95cacb5cd894ba84277d418343d47f0303505ef132b69676465d909f6e3: Status 404 returned error can't find the container with id 5c52d95cacb5cd894ba84277d418343d47f0303505ef132b69676465d909f6e3 Jul 17 17:38:42.931772 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:42.931737 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" event={"ID":"389f5ad4-5a3e-483f-90da-4b80791d0f0c","Type":"ContainerStarted","Data":"5c52d95cacb5cd894ba84277d418343d47f0303505ef132b69676465d909f6e3"} Jul 17 17:38:43.938261 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:43.938225 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" event={"ID":"389f5ad4-5a3e-483f-90da-4b80791d0f0c","Type":"ContainerStarted","Data":"8a5ae32dbd29af8fd361fb3b99b9cb43c1f636c237e2fa87fe58b4fb8d628d1f"} Jul 17 17:38:43.938690 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:43.938319 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" Jul 17 17:38:43.958871 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:43.958821 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" podStartSLOduration=1.4951421169999999 podStartE2EDuration="1.958803983s" totalImagesPullingTime="463.661866ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:38:43.955354791 +0000 UTC m=+844.690778850" watchObservedRunningTime="2026-07-17 17:38:43.958803983 +0000 UTC m=+844.694228031" Jul 17 17:38:54.944479 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:54.944445 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" Jul 17 17:38:59.457757 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:59.457721 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:59.563753 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:59.563720 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54f8d57c9d-xxmrr"] Jul 17 17:38:59.563986 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:59.563961 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" podUID="389f5ad4-5a3e-483f-90da-4b80791d0f0c" containerName="manager" containerID="cri-o://8a5ae32dbd29af8fd361fb3b99b9cb43c1f636c237e2fa87fe58b4fb8d628d1f" gracePeriod=30 Jul 17 17:38:59.809996 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:59.809969 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" Jul 17 17:38:59.875005 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:59.874978 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm886\" (UniqueName: \"kubernetes.io/projected/389f5ad4-5a3e-483f-90da-4b80791d0f0c-kube-api-access-qm886\") pod \"389f5ad4-5a3e-483f-90da-4b80791d0f0c\" (UID: \"389f5ad4-5a3e-483f-90da-4b80791d0f0c\") " Jul 17 17:38:59.875174 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:59.875091 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/389f5ad4-5a3e-483f-90da-4b80791d0f0c-webhook-cert\") pod \"389f5ad4-5a3e-483f-90da-4b80791d0f0c\" (UID: \"389f5ad4-5a3e-483f-90da-4b80791d0f0c\") " Jul 17 17:38:59.877167 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:59.877139 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/389f5ad4-5a3e-483f-90da-4b80791d0f0c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "389f5ad4-5a3e-483f-90da-4b80791d0f0c" (UID: "389f5ad4-5a3e-483f-90da-4b80791d0f0c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:38:59.877399 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:59.877382 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389f5ad4-5a3e-483f-90da-4b80791d0f0c-kube-api-access-qm886" (OuterVolumeSpecName: "kube-api-access-qm886") pod "389f5ad4-5a3e-483f-90da-4b80791d0f0c" (UID: "389f5ad4-5a3e-483f-90da-4b80791d0f0c"). InnerVolumeSpecName "kube-api-access-qm886". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:38:59.976352 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:59.976323 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm886\" (UniqueName: \"kubernetes.io/projected/389f5ad4-5a3e-483f-90da-4b80791d0f0c-kube-api-access-qm886\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:38:59.976352 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:38:59.976348 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/389f5ad4-5a3e-483f-90da-4b80791d0f0c-webhook-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:39:00.014631 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.014595 2573 generic.go:356] "Generic (PLEG): container finished" podID="389f5ad4-5a3e-483f-90da-4b80791d0f0c" containerID="8a5ae32dbd29af8fd361fb3b99b9cb43c1f636c237e2fa87fe58b4fb8d628d1f" exitCode=0 Jul 17 17:39:00.014799 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.014668 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" Jul 17 17:39:00.014799 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.014684 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" event={"ID":"389f5ad4-5a3e-483f-90da-4b80791d0f0c","Type":"ContainerDied","Data":"8a5ae32dbd29af8fd361fb3b99b9cb43c1f636c237e2fa87fe58b4fb8d628d1f"} Jul 17 17:39:00.014799 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.014735 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54f8d57c9d-xxmrr" event={"ID":"389f5ad4-5a3e-483f-90da-4b80791d0f0c","Type":"ContainerDied","Data":"5c52d95cacb5cd894ba84277d418343d47f0303505ef132b69676465d909f6e3"} Jul 17 17:39:00.014799 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.014758 2573 scope.go:117] "RemoveContainer" containerID="8a5ae32dbd29af8fd361fb3b99b9cb43c1f636c237e2fa87fe58b4fb8d628d1f" Jul 17 17:39:00.023485 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.023464 2573 scope.go:117] "RemoveContainer" containerID="8a5ae32dbd29af8fd361fb3b99b9cb43c1f636c237e2fa87fe58b4fb8d628d1f" Jul 17 17:39:00.023781 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:39:00.023763 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5ae32dbd29af8fd361fb3b99b9cb43c1f636c237e2fa87fe58b4fb8d628d1f\": container with ID starting with 8a5ae32dbd29af8fd361fb3b99b9cb43c1f636c237e2fa87fe58b4fb8d628d1f not found: ID does not exist" containerID="8a5ae32dbd29af8fd361fb3b99b9cb43c1f636c237e2fa87fe58b4fb8d628d1f" Jul 17 17:39:00.023833 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.023789 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5ae32dbd29af8fd361fb3b99b9cb43c1f636c237e2fa87fe58b4fb8d628d1f"} err="failed to get container status \"8a5ae32dbd29af8fd361fb3b99b9cb43c1f636c237e2fa87fe58b4fb8d628d1f\": rpc error: code = NotFound desc = could not find container \"8a5ae32dbd29af8fd361fb3b99b9cb43c1f636c237e2fa87fe58b4fb8d628d1f\": container with ID starting with 8a5ae32dbd29af8fd361fb3b99b9cb43c1f636c237e2fa87fe58b4fb8d628d1f not found: ID does not exist" Jul 17 17:39:00.036719 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.036694 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54f8d57c9d-xxmrr"] Jul 17 17:39:00.040661 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.040639 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-54f8d57c9d-xxmrr"] Jul 17 17:39:00.706785 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.706751 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:00.823152 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.823116 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-54f8d57c9d-hp6hv"] Jul 17 17:39:00.823594 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.823578 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="389f5ad4-5a3e-483f-90da-4b80791d0f0c" containerName="manager" Jul 17 17:39:00.823594 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.823594 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="389f5ad4-5a3e-483f-90da-4b80791d0f0c" containerName="manager" Jul 17 17:39:00.823746 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.823684 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="389f5ad4-5a3e-483f-90da-4b80791d0f0c" containerName="manager" Jul 17 17:39:00.828114 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.828095 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54f8d57c9d-hp6hv" Jul 17 17:39:00.830849 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.830830 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 17 17:39:00.831686 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.831669 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 17 17:39:00.831775 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.831671 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vx4rq\"" Jul 17 17:39:00.838935 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.838915 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54f8d57c9d-hp6hv"] Jul 17 17:39:00.885730 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.885700 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76e8c837-dd34-4774-92c4-347adb3ca1a6-webhook-cert\") pod \"maas-controller-54f8d57c9d-hp6hv\" (UID: \"76e8c837-dd34-4774-92c4-347adb3ca1a6\") " pod="opendatahub/maas-controller-54f8d57c9d-hp6hv" Jul 17 17:39:00.885868 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.885734 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzcct\" (UniqueName: \"kubernetes.io/projected/76e8c837-dd34-4774-92c4-347adb3ca1a6-kube-api-access-dzcct\") pod \"maas-controller-54f8d57c9d-hp6hv\" (UID: \"76e8c837-dd34-4774-92c4-347adb3ca1a6\") " pod="opendatahub/maas-controller-54f8d57c9d-hp6hv" Jul 17 17:39:00.987331 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.987298 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76e8c837-dd34-4774-92c4-347adb3ca1a6-webhook-cert\") pod \"maas-controller-54f8d57c9d-hp6hv\" (UID: \"76e8c837-dd34-4774-92c4-347adb3ca1a6\") " pod="opendatahub/maas-controller-54f8d57c9d-hp6hv" Jul 17 17:39:00.987473 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.987338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzcct\" (UniqueName: \"kubernetes.io/projected/76e8c837-dd34-4774-92c4-347adb3ca1a6-kube-api-access-dzcct\") pod \"maas-controller-54f8d57c9d-hp6hv\" (UID: \"76e8c837-dd34-4774-92c4-347adb3ca1a6\") " pod="opendatahub/maas-controller-54f8d57c9d-hp6hv" Jul 17 17:39:00.989697 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.989670 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76e8c837-dd34-4774-92c4-347adb3ca1a6-webhook-cert\") pod \"maas-controller-54f8d57c9d-hp6hv\" (UID: \"76e8c837-dd34-4774-92c4-347adb3ca1a6\") " pod="opendatahub/maas-controller-54f8d57c9d-hp6hv" Jul 17 17:39:00.998442 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:00.998410 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzcct\" (UniqueName: \"kubernetes.io/projected/76e8c837-dd34-4774-92c4-347adb3ca1a6-kube-api-access-dzcct\") pod \"maas-controller-54f8d57c9d-hp6hv\" (UID: \"76e8c837-dd34-4774-92c4-347adb3ca1a6\") " pod="opendatahub/maas-controller-54f8d57c9d-hp6hv" Jul 17 17:39:01.139743 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:01.139706 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54f8d57c9d-hp6hv" Jul 17 17:39:01.270455 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:01.270376 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54f8d57c9d-hp6hv"] Jul 17 17:39:01.273379 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:39:01.273342 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e8c837_dd34_4774_92c4_347adb3ca1a6.slice/crio-ae27c36e7a7c92f8bb17d888a8126c7144ad4b7c8b901e1058a24d45cb2117d4 WatchSource:0}: Error finding container ae27c36e7a7c92f8bb17d888a8126c7144ad4b7c8b901e1058a24d45cb2117d4: Status 404 returned error can't find the container with id ae27c36e7a7c92f8bb17d888a8126c7144ad4b7c8b901e1058a24d45cb2117d4 Jul 17 17:39:01.784070 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:01.784034 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="389f5ad4-5a3e-483f-90da-4b80791d0f0c" path="/var/lib/kubelet/pods/389f5ad4-5a3e-483f-90da-4b80791d0f0c/volumes" Jul 17 17:39:02.026638 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:02.026584 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54f8d57c9d-hp6hv" event={"ID":"76e8c837-dd34-4774-92c4-347adb3ca1a6","Type":"ContainerStarted","Data":"f5699fac6942a0f61f6e16757c91cb086045cea7b4de9ba3b0e335f42f3f3c6e"} Jul 17 17:39:02.026786 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:02.026657 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54f8d57c9d-hp6hv" event={"ID":"76e8c837-dd34-4774-92c4-347adb3ca1a6","Type":"ContainerStarted","Data":"ae27c36e7a7c92f8bb17d888a8126c7144ad4b7c8b901e1058a24d45cb2117d4"} Jul 17 17:39:02.026786 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:02.026757 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-54f8d57c9d-hp6hv" Jul 17 17:39:02.046925 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:02.046873 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-54f8d57c9d-hp6hv" podStartSLOduration=1.725276169 podStartE2EDuration="2.046860938s" totalImagesPullingTime="321.584769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:39:02.04514456 +0000 UTC m=+862.780568604" watchObservedRunningTime="2026-07-17 17:39:02.046860938 +0000 UTC m=+862.782284986" Jul 17 17:39:13.033720 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:13.033686 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-54f8d57c9d-hp6hv" Jul 17 17:39:39.716631 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:39.716582 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:39:39.717111 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:39.717095 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:39:41.357479 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:41.357445 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:42.654117 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:42.654080 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:47.635255 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:47.635216 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:47.744362 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:47.744333 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:47.857840 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:47.857806 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:48.046275 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:48.046243 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:53.822480 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:53.822441 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:53.923655 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:39:53.923602 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:41:33.120732 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:41:33.120697 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:42:21.901509 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:42:21.901458 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:43:11.070866 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:43:11.070787 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:43:11.131548 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:43:11.131514 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:44:39.749927 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:44:39.749898 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:44:39.751128 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:44:39.751103 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:48:40.523573 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:48:40.523480 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:48:59.946400 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:48:59.946359 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:49:39.782031 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:49:39.781994 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:49:39.784400 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:49:39.784331 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:49:50.311700 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:49:50.311663 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:49:50.412483 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:49:50.412449 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:00.415249 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:50:00.415210 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:51.680350 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:50:51.680305 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:59.152589 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:50:59.152544 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:07.429573 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:07.429531 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:12.591099 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:12.591046 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:15.728594 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:15.728553 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:15.821554 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:15.821523 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:16.229211 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:16.229169 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:22.262735 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.262686 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:22.323686 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.323655 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:22.462182 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.462157 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh"] Jul 17 17:51:22.465734 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.465704 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.468455 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.468434 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ce07281e-openshift-default-dockercfg-h9m9k\"" Jul 17 17:51:22.503930 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.503892 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:22.526350 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.526277 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh"] Jul 17 17:51:22.560036 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.560004 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd185d4-27ae-4987-9176-e59138a35c46-istiod-ca-cert\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.560179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.560049 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-workload-socket\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.560179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.560121 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkxch\" (UniqueName: \"kubernetes.io/projected/8bd185d4-27ae-4987-9176-e59138a35c46-kube-api-access-rkxch\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.560179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.560139 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-istio-envoy\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.560179 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.560170 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd185d4-27ae-4987-9176-e59138a35c46-istio-token\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.560384 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.560201 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-credential-socket\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.560384 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.560319 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-workload-certs\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.560384 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.560366 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-istio-data\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.560520 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.560402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd185d4-27ae-4987-9176-e59138a35c46-istio-podinfo\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.604715 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.604682 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:22.661355 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.661323 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-credential-socket\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.661527 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.661375 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-workload-certs\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.661527 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.661400 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-istio-data\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.661527 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.661428 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd185d4-27ae-4987-9176-e59138a35c46-istio-podinfo\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.661527 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.661502 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd185d4-27ae-4987-9176-e59138a35c46-istiod-ca-cert\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.661788 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.661540 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-workload-socket\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.661788 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.661609 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkxch\" (UniqueName: \"kubernetes.io/projected/8bd185d4-27ae-4987-9176-e59138a35c46-kube-api-access-rkxch\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.661788 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.661666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-istio-envoy\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.661788 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.661707 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd185d4-27ae-4987-9176-e59138a35c46-istio-token\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.661788 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.661770 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-credential-socket\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.662017 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.661808 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-workload-certs\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.662017 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.661949 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-workload-socket\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.662202 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.662182 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-istio-data\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.662298 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.662281 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd185d4-27ae-4987-9176-e59138a35c46-istiod-ca-cert\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.663920 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.663901 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-istio-envoy\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.664028 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.664010 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd185d4-27ae-4987-9176-e59138a35c46-istio-podinfo\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.670058 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.670035 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd185d4-27ae-4987-9176-e59138a35c46-istio-token\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.670167 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.670091 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkxch\" (UniqueName: \"kubernetes.io/projected/8bd185d4-27ae-4987-9176-e59138a35c46-kube-api-access-rkxch\") pod \"e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.775916 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.775890 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:22.917289 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.917260 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh"] Jul 17 17:51:22.919148 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:51:22.919124 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd185d4_27ae_4987_9176_e59138a35c46.slice/crio-40c1eb520eaf5d384992f9b1cbeea1c32d86995288b871e3fd31f920daf101a7 WatchSource:0}: Error finding container 40c1eb520eaf5d384992f9b1cbeea1c32d86995288b871e3fd31f920daf101a7: Status 404 returned error can't find the container with id 40c1eb520eaf5d384992f9b1cbeea1c32d86995288b871e3fd31f920daf101a7 Jul 17 17:51:22.921587 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.921558 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:51:22.921682 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.921634 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:51:22.921682 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:22.921667 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:51:23.688944 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.688908 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:23.736138 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.736107 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" event={"ID":"8bd185d4-27ae-4987-9176-e59138a35c46","Type":"ContainerStarted","Data":"5ada1029c61544330fd5e6c6bb478d6f3314fcaee34cfa91cb475de78ab926f0"} Jul 17 17:51:23.736311 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.736164 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" event={"ID":"8bd185d4-27ae-4987-9176-e59138a35c46","Type":"ContainerStarted","Data":"40c1eb520eaf5d384992f9b1cbeea1c32d86995288b871e3fd31f920daf101a7"} Jul 17 17:51:23.761080 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.761028 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" podStartSLOduration=1.761006321 podStartE2EDuration="1.761006321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:51:23.759526374 +0000 UTC m=+1604.494950442" watchObservedRunningTime="2026-07-17 17:51:23.761006321 +0000 UTC m=+1604.496430371" Jul 17 17:51:23.778953 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.778926 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:23.780330 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.780295 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:23.781653 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.781633 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:23.786413 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.786391 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx"] Jul 17 17:51:23.795274 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.795251 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:51:23.797997 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.797826 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ce07281e-serving-cert\"" Jul 17 17:51:23.801403 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.801383 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx"] Jul 17 17:51:23.858294 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.858249 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:23.876232 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.876190 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-openshift-service-ca\") pod \"maas-api-e2e-ait-ce07281e-55644d4946-mr4bx\" (UID: \"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:51:23.876426 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.876303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkbbn\" (UniqueName: \"kubernetes.io/projected/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-kube-api-access-tkbbn\") pod \"maas-api-e2e-ait-ce07281e-55644d4946-mr4bx\" (UID: \"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:51:23.876426 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.876362 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-maas-api-tls\") pod \"maas-api-e2e-ait-ce07281e-55644d4946-mr4bx\" (UID: \"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:51:23.957941 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.957831 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:23.977354 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.977321 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-openshift-service-ca\") pod \"maas-api-e2e-ait-ce07281e-55644d4946-mr4bx\" (UID: \"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:51:23.977518 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.977376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkbbn\" (UniqueName: \"kubernetes.io/projected/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-kube-api-access-tkbbn\") pod \"maas-api-e2e-ait-ce07281e-55644d4946-mr4bx\" (UID: \"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:51:23.977518 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.977407 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-maas-api-tls\") pod \"maas-api-e2e-ait-ce07281e-55644d4946-mr4bx\" (UID: \"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:51:23.978010 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.977984 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-openshift-service-ca\") pod \"maas-api-e2e-ait-ce07281e-55644d4946-mr4bx\" (UID: \"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:51:23.979763 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.979744 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-maas-api-tls\") pod \"maas-api-e2e-ait-ce07281e-55644d4946-mr4bx\" (UID: \"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:51:23.986360 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:23.986336 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkbbn\" (UniqueName: \"kubernetes.io/projected/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-kube-api-access-tkbbn\") pod \"maas-api-e2e-ait-ce07281e-55644d4946-mr4bx\" (UID: \"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:51:24.108413 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:24.108383 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:51:24.451313 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:24.451279 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:24.473978 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:24.473954 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx"] Jul 17 17:51:24.476414 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:51:24.476384 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9b2850_f0ff_4ad2_a5eb_115eb8d0f5ce.slice/crio-cec5abf2a8bc07845a92643a0ccea5ccfc1d5a9af5f724260a048a537b41d428 WatchSource:0}: Error finding container cec5abf2a8bc07845a92643a0ccea5ccfc1d5a9af5f724260a048a537b41d428: Status 404 returned error can't find the container with id cec5abf2a8bc07845a92643a0ccea5ccfc1d5a9af5f724260a048a537b41d428 Jul 17 17:51:24.477984 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:24.477964 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 17:51:24.590594 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:24.590564 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:24.742639 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:24.742593 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" event={"ID":"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce","Type":"ContainerStarted","Data":"cec5abf2a8bc07845a92643a0ccea5ccfc1d5a9af5f724260a048a537b41d428"} Jul 17 17:51:24.743075 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:24.742908 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:24.743810 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:24.743792 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:25.748402 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:25.748364 2573 generic.go:356] "Generic (PLEG): container finished" podID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerID="0f04bbc59af8cb066e49be7be4d68b5eedf88a6118d360be9b377663029c9297" exitCode=1 Jul 17 17:51:25.748828 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:25.748458 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" event={"ID":"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce","Type":"ContainerDied","Data":"0f04bbc59af8cb066e49be7be4d68b5eedf88a6118d360be9b377663029c9297"} Jul 17 17:51:25.748828 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:25.748688 2573 scope.go:117] "RemoveContainer" containerID="0f04bbc59af8cb066e49be7be4d68b5eedf88a6118d360be9b377663029c9297" Jul 17 17:51:26.452270 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:26.452239 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:26.755703 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:26.755669 2573 generic.go:356] "Generic (PLEG): container finished" podID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerID="a983e5e8950886c2bfd29ee5a0c0cf727b58c18e6fc201ce099335e0b13c08ae" exitCode=1 Jul 17 17:51:26.756150 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:26.755748 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" event={"ID":"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce","Type":"ContainerDied","Data":"a983e5e8950886c2bfd29ee5a0c0cf727b58c18e6fc201ce099335e0b13c08ae"} Jul 17 17:51:26.756150 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:26.755801 2573 scope.go:117] "RemoveContainer" containerID="0f04bbc59af8cb066e49be7be4d68b5eedf88a6118d360be9b377663029c9297" Jul 17 17:51:26.756150 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:26.756031 2573 scope.go:117] "RemoveContainer" containerID="a983e5e8950886c2bfd29ee5a0c0cf727b58c18e6fc201ce099335e0b13c08ae" Jul 17 17:51:26.756328 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:51:26.756271 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:51:27.763765 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:27.763734 2573 scope.go:117] "RemoveContainer" containerID="a983e5e8950886c2bfd29ee5a0c0cf727b58c18e6fc201ce099335e0b13c08ae" Jul 17 17:51:27.764207 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:51:27.763976 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:51:29.108766 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:29.108725 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:51:29.109912 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:29.109891 2573 scope.go:117] "RemoveContainer" containerID="a983e5e8950886c2bfd29ee5a0c0cf727b58c18e6fc201ce099335e0b13c08ae" Jul 17 17:51:29.110749 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:51:29.110315 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:51:34.109298 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:34.109264 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:51:34.109696 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:34.109639 2573 scope.go:117] "RemoveContainer" containerID="a983e5e8950886c2bfd29ee5a0c0cf727b58c18e6fc201ce099335e0b13c08ae" Jul 17 17:51:34.109834 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:51:34.109813 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:51:35.804997 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:35.804960 2573 generic.go:356] "Generic (PLEG): container finished" podID="8bd185d4-27ae-4987-9176-e59138a35c46" containerID="5ada1029c61544330fd5e6c6bb478d6f3314fcaee34cfa91cb475de78ab926f0" exitCode=0 Jul 17 17:51:35.805433 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:35.805031 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" event={"ID":"8bd185d4-27ae-4987-9176-e59138a35c46","Type":"ContainerDied","Data":"5ada1029c61544330fd5e6c6bb478d6f3314fcaee34cfa91cb475de78ab926f0"} Jul 17 17:51:35.805433 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:35.805401 2573 scope.go:117] "RemoveContainer" containerID="5ada1029c61544330fd5e6c6bb478d6f3314fcaee34cfa91cb475de78ab926f0" Jul 17 17:51:35.806341 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:35.806305 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:51:35.806470 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:35.806393 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:51:35.806470 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:35.806441 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:51:36.811677 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:36.811641 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" event={"ID":"8bd185d4-27ae-4987-9176-e59138a35c46","Type":"ContainerStarted","Data":"b0ad2fae034bac74a30d27e20446ccce538e15fb563926dabcfa3ea2812da9a3"} Jul 17 17:51:37.777145 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:37.777111 2573 patch_prober.go:28] interesting pod/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 17 17:51:37.777330 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:37.777178 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" podUID="8bd185d4-27ae-4987-9176-e59138a35c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 17 17:51:37.778482 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:37.778463 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:38.777051 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:38.777023 2573 patch_prober.go:28] interesting pod/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 17 17:51:38.777435 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:38.777071 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" podUID="8bd185d4-27ae-4987-9176-e59138a35c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 17 17:51:39.743769 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:39.743734 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:39.777012 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:39.776980 2573 patch_prober.go:28] interesting pod/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 17 17:51:39.777145 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:39.777040 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" podUID="8bd185d4-27ae-4987-9176-e59138a35c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 17 17:51:40.777284 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:40.777246 2573 patch_prober.go:28] interesting pod/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 17 17:51:40.777707 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:40.777299 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" podUID="8bd185d4-27ae-4987-9176-e59138a35c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 17 17:51:41.780893 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:41.780866 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:41.781851 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:41.781825 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:51:44.772351 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:44.772316 2573 scope.go:117] "RemoveContainer" containerID="a983e5e8950886c2bfd29ee5a0c0cf727b58c18e6fc201ce099335e0b13c08ae" Jul 17 17:51:45.866854 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:45.866753 2573 generic.go:356] "Generic (PLEG): container finished" podID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerID="01f67c0fb95d9de4fe6e48df65c11ed3eef8a2cef5df48ecc0d3fa77bbe11115" exitCode=1 Jul 17 17:51:45.866854 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:45.866813 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" event={"ID":"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce","Type":"ContainerDied","Data":"01f67c0fb95d9de4fe6e48df65c11ed3eef8a2cef5df48ecc0d3fa77bbe11115"} Jul 17 17:51:45.867332 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:45.866875 2573 scope.go:117] "RemoveContainer" containerID="a983e5e8950886c2bfd29ee5a0c0cf727b58c18e6fc201ce099335e0b13c08ae" Jul 17 17:51:45.867332 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:45.867226 2573 scope.go:117] "RemoveContainer" containerID="01f67c0fb95d9de4fe6e48df65c11ed3eef8a2cef5df48ecc0d3fa77bbe11115" Jul 17 17:51:45.867494 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:51:45.867472 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:51:49.108682 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:49.108640 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:51:49.109076 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:49.109044 2573 scope.go:117] "RemoveContainer" containerID="01f67c0fb95d9de4fe6e48df65c11ed3eef8a2cef5df48ecc0d3fa77bbe11115" Jul 17 17:51:49.109254 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:51:49.109234 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:51:54.108675 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:54.108607 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:51:54.109157 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:51:54.109026 2573 scope.go:117] "RemoveContainer" containerID="01f67c0fb95d9de4fe6e48df65c11ed3eef8a2cef5df48ecc0d3fa77bbe11115" Jul 17 17:51:54.109219 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:51:54.109199 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:52:06.772031 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:06.771998 2573 scope.go:117] "RemoveContainer" containerID="01f67c0fb95d9de4fe6e48df65c11ed3eef8a2cef5df48ecc0d3fa77bbe11115" Jul 17 17:52:07.980055 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:07.979964 2573 generic.go:356] "Generic (PLEG): container finished" podID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerID="b9a3abdcae0ed2316df8566f86252f30f5bafbbf674a79ad106ae516dfcd4db1" exitCode=1 Jul 17 17:52:07.980055 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:07.980000 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" event={"ID":"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce","Type":"ContainerDied","Data":"b9a3abdcae0ed2316df8566f86252f30f5bafbbf674a79ad106ae516dfcd4db1"} Jul 17 17:52:07.980569 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:07.980064 2573 scope.go:117] "RemoveContainer" containerID="01f67c0fb95d9de4fe6e48df65c11ed3eef8a2cef5df48ecc0d3fa77bbe11115" Jul 17 17:52:07.980569 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:07.980449 2573 scope.go:117] "RemoveContainer" containerID="b9a3abdcae0ed2316df8566f86252f30f5bafbbf674a79ad106ae516dfcd4db1" Jul 17 17:52:07.980744 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:52:07.980723 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:52:09.109354 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:09.109320 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:52:09.109874 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:09.109857 2573 scope.go:117] "RemoveContainer" containerID="b9a3abdcae0ed2316df8566f86252f30f5bafbbf674a79ad106ae516dfcd4db1" Jul 17 17:52:09.110101 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:52:09.110077 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:52:14.109099 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:14.109063 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:52:14.109595 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:14.109549 2573 scope.go:117] "RemoveContainer" containerID="b9a3abdcae0ed2316df8566f86252f30f5bafbbf674a79ad106ae516dfcd4db1" Jul 17 17:52:14.109827 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:52:14.109803 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:52:26.772029 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:26.771995 2573 scope.go:117] "RemoveContainer" containerID="b9a3abdcae0ed2316df8566f86252f30f5bafbbf674a79ad106ae516dfcd4db1" Jul 17 17:52:26.772434 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:52:26.772195 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:52:41.772336 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:41.772299 2573 scope.go:117] "RemoveContainer" containerID="b9a3abdcae0ed2316df8566f86252f30f5bafbbf674a79ad106ae516dfcd4db1" Jul 17 17:52:41.772933 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:52:41.772527 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:52:54.771745 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:54.771706 2573 scope.go:117] "RemoveContainer" containerID="b9a3abdcae0ed2316df8566f86252f30f5bafbbf674a79ad106ae516dfcd4db1" Jul 17 17:52:56.228102 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:56.228070 2573 generic.go:356] "Generic (PLEG): container finished" podID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerID="c495b4265443e8ada1f604dee823f5319cb1bb8c54331a3c9442e3fcb683ed54" exitCode=1 Jul 17 17:52:56.228498 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:56.228147 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" event={"ID":"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce","Type":"ContainerDied","Data":"c495b4265443e8ada1f604dee823f5319cb1bb8c54331a3c9442e3fcb683ed54"} Jul 17 17:52:56.228498 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:56.228209 2573 scope.go:117] "RemoveContainer" containerID="b9a3abdcae0ed2316df8566f86252f30f5bafbbf674a79ad106ae516dfcd4db1" Jul 17 17:52:56.228612 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:56.228523 2573 scope.go:117] "RemoveContainer" containerID="c495b4265443e8ada1f604dee823f5319cb1bb8c54331a3c9442e3fcb683ed54" Jul 17 17:52:56.228768 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:52:56.228749 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:52:59.109040 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:59.109002 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:52:59.109463 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:52:59.109411 2573 scope.go:117] "RemoveContainer" containerID="c495b4265443e8ada1f604dee823f5319cb1bb8c54331a3c9442e3fcb683ed54" Jul 17 17:52:59.109604 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:52:59.109584 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:53:04.108755 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:53:04.108676 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:53:04.109198 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:53:04.109008 2573 scope.go:117] "RemoveContainer" containerID="c495b4265443e8ada1f604dee823f5319cb1bb8c54331a3c9442e3fcb683ed54" Jul 17 17:53:04.109198 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:53:04.109178 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:53:17.771982 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:53:17.771946 2573 scope.go:117] "RemoveContainer" containerID="c495b4265443e8ada1f604dee823f5319cb1bb8c54331a3c9442e3fcb683ed54" Jul 17 17:53:17.772357 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:53:17.772146 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:53:29.774597 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:53:29.774570 2573 scope.go:117] "RemoveContainer" containerID="c495b4265443e8ada1f604dee823f5319cb1bb8c54331a3c9442e3fcb683ed54" Jul 17 17:53:29.775110 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:53:29.774782 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:53:43.771559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:53:43.771525 2573 scope.go:117] "RemoveContainer" containerID="c495b4265443e8ada1f604dee823f5319cb1bb8c54331a3c9442e3fcb683ed54" Jul 17 17:53:43.772016 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:53:43.771750 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:53:56.771737 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:53:56.771707 2573 scope.go:117] "RemoveContainer" containerID="c495b4265443e8ada1f604dee823f5319cb1bb8c54331a3c9442e3fcb683ed54" Jul 17 17:53:56.772121 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:53:56.771940 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:54:07.771890 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:07.771860 2573 scope.go:117] "RemoveContainer" containerID="c495b4265443e8ada1f604dee823f5319cb1bb8c54331a3c9442e3fcb683ed54" Jul 17 17:54:07.772313 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:54:07.772057 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:54:18.771992 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:18.771960 2573 scope.go:117] "RemoveContainer" containerID="c495b4265443e8ada1f604dee823f5319cb1bb8c54331a3c9442e3fcb683ed54" Jul 17 17:54:19.643311 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:19.643272 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" event={"ID":"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce","Type":"ContainerStarted","Data":"f6a0a63a4322c314bfb6f583c4c4d8f41cb1861e7dc974f40468bd0dcf1259ed"} Jul 17 17:54:19.643529 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:19.643510 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:54:19.665214 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:19.665114 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podStartSLOduration=173.733991377 podStartE2EDuration="2m56.665099039s" totalImagesPullingTime="2.931107662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:51:23 +0000 UTC" imagePullSessionsCount=6 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:54:19.663755815 +0000 UTC m=+1780.399179894" watchObservedRunningTime="2026-07-17 17:54:19.665099039 +0000 UTC m=+1780.400523087" Jul 17 17:54:20.650230 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:20.650193 2573 generic.go:356] "Generic (PLEG): container finished" podID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerID="f6a0a63a4322c314bfb6f583c4c4d8f41cb1861e7dc974f40468bd0dcf1259ed" exitCode=1 Jul 17 17:54:20.650651 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:20.650274 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" event={"ID":"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce","Type":"ContainerDied","Data":"f6a0a63a4322c314bfb6f583c4c4d8f41cb1861e7dc974f40468bd0dcf1259ed"} Jul 17 17:54:20.650651 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:20.650324 2573 scope.go:117] "RemoveContainer" containerID="c495b4265443e8ada1f604dee823f5319cb1bb8c54331a3c9442e3fcb683ed54" Jul 17 17:54:20.650651 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:20.650559 2573 scope.go:117] "RemoveContainer" containerID="f6a0a63a4322c314bfb6f583c4c4d8f41cb1861e7dc974f40468bd0dcf1259ed" Jul 17 17:54:20.650818 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:54:20.650797 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:54:21.658286 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:21.658261 2573 scope.go:117] "RemoveContainer" containerID="f6a0a63a4322c314bfb6f583c4c4d8f41cb1861e7dc974f40468bd0dcf1259ed" Jul 17 17:54:21.658671 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:54:21.658448 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:54:23.829308 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:23.829267 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:24.109681 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:24.109576 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:54:24.110026 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:24.110012 2573 scope.go:117] "RemoveContainer" containerID="f6a0a63a4322c314bfb6f583c4c4d8f41cb1861e7dc974f40468bd0dcf1259ed" Jul 17 17:54:24.110238 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:54:24.110218 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-ce07281e-55644d4946-mr4bx_odh-ai-gateway-infra(2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" Jul 17 17:54:25.273270 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:25.273224 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:28.829587 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:28.829554 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh"] Jul 17 17:54:28.830013 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:28.829783 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" podUID="8bd185d4-27ae-4987-9176-e59138a35c46" containerName="istio-proxy" containerID="cri-o://b0ad2fae034bac74a30d27e20446ccce538e15fb563926dabcfa3ea2812da9a3" gracePeriod=30 Jul 17 17:54:28.863635 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:28.863597 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:28.960793 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:28.960760 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:34.080651 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.080615 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:54:34.257084 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.257054 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkxch\" (UniqueName: \"kubernetes.io/projected/8bd185d4-27ae-4987-9176-e59138a35c46-kube-api-access-rkxch\") pod \"8bd185d4-27ae-4987-9176-e59138a35c46\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " Jul 17 17:54:34.257250 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.257091 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd185d4-27ae-4987-9176-e59138a35c46-istio-podinfo\") pod \"8bd185d4-27ae-4987-9176-e59138a35c46\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " Jul 17 17:54:34.257250 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.257131 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd185d4-27ae-4987-9176-e59138a35c46-istio-token\") pod \"8bd185d4-27ae-4987-9176-e59138a35c46\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " Jul 17 17:54:34.257250 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.257167 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-istio-envoy\") pod \"8bd185d4-27ae-4987-9176-e59138a35c46\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " Jul 17 17:54:34.257250 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.257185 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-workload-certs\") pod \"8bd185d4-27ae-4987-9176-e59138a35c46\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " Jul 17 17:54:34.257250 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.257210 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-workload-socket\") pod \"8bd185d4-27ae-4987-9176-e59138a35c46\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " Jul 17 17:54:34.257250 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.257238 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-istio-data\") pod \"8bd185d4-27ae-4987-9176-e59138a35c46\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " Jul 17 17:54:34.257570 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.257280 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd185d4-27ae-4987-9176-e59138a35c46-istiod-ca-cert\") pod \"8bd185d4-27ae-4987-9176-e59138a35c46\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " Jul 17 17:54:34.257570 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.257369 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-credential-socket\") pod \"8bd185d4-27ae-4987-9176-e59138a35c46\" (UID: \"8bd185d4-27ae-4987-9176-e59138a35c46\") " Jul 17 17:54:34.257570 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.257502 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8bd185d4-27ae-4987-9176-e59138a35c46" (UID: "8bd185d4-27ae-4987-9176-e59138a35c46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:54:34.257570 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.257545 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8bd185d4-27ae-4987-9176-e59138a35c46" (UID: "8bd185d4-27ae-4987-9176-e59138a35c46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:54:34.257832 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.257765 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:54:34.257832 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.257783 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:54:34.257832 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.257779 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8bd185d4-27ae-4987-9176-e59138a35c46" (UID: "8bd185d4-27ae-4987-9176-e59138a35c46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:54:34.257832 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.257800 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd185d4-27ae-4987-9176-e59138a35c46-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8bd185d4-27ae-4987-9176-e59138a35c46" (UID: "8bd185d4-27ae-4987-9176-e59138a35c46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:54:34.258560 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.258531 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-istio-data" (OuterVolumeSpecName: "istio-data") pod "8bd185d4-27ae-4987-9176-e59138a35c46" (UID: "8bd185d4-27ae-4987-9176-e59138a35c46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:54:34.259542 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.259514 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bd185d4-27ae-4987-9176-e59138a35c46-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8bd185d4-27ae-4987-9176-e59138a35c46" (UID: "8bd185d4-27ae-4987-9176-e59138a35c46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 17:54:34.259542 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.259525 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd185d4-27ae-4987-9176-e59138a35c46-kube-api-access-rkxch" (OuterVolumeSpecName: "kube-api-access-rkxch") pod "8bd185d4-27ae-4987-9176-e59138a35c46" (UID: "8bd185d4-27ae-4987-9176-e59138a35c46"). InnerVolumeSpecName "kube-api-access-rkxch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:54:34.259542 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.259533 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd185d4-27ae-4987-9176-e59138a35c46-istio-token" (OuterVolumeSpecName: "istio-token") pod "8bd185d4-27ae-4987-9176-e59138a35c46" (UID: "8bd185d4-27ae-4987-9176-e59138a35c46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:54:34.259741 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.259528 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8bd185d4-27ae-4987-9176-e59138a35c46" (UID: "8bd185d4-27ae-4987-9176-e59138a35c46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:54:34.358824 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.358791 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:54:34.358824 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.358824 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkxch\" (UniqueName: \"kubernetes.io/projected/8bd185d4-27ae-4987-9176-e59138a35c46-kube-api-access-rkxch\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:54:34.359044 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.358837 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd185d4-27ae-4987-9176-e59138a35c46-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:54:34.359044 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.358853 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd185d4-27ae-4987-9176-e59138a35c46-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:54:34.359044 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.358864 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:54:34.359044 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.358874 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd185d4-27ae-4987-9176-e59138a35c46-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:54:34.359044 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.358885 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd185d4-27ae-4987-9176-e59138a35c46-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:54:34.722153 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.722118 2573 generic.go:356] "Generic (PLEG): container finished" podID="8bd185d4-27ae-4987-9176-e59138a35c46" containerID="b0ad2fae034bac74a30d27e20446ccce538e15fb563926dabcfa3ea2812da9a3" exitCode=0 Jul 17 17:54:34.722326 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.722164 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" Jul 17 17:54:34.722326 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.722202 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" event={"ID":"8bd185d4-27ae-4987-9176-e59138a35c46","Type":"ContainerDied","Data":"b0ad2fae034bac74a30d27e20446ccce538e15fb563926dabcfa3ea2812da9a3"} Jul 17 17:54:34.722326 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.722260 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh" event={"ID":"8bd185d4-27ae-4987-9176-e59138a35c46","Type":"ContainerDied","Data":"40c1eb520eaf5d384992f9b1cbeea1c32d86995288b871e3fd31f920daf101a7"} Jul 17 17:54:34.722326 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.722283 2573 scope.go:117] "RemoveContainer" containerID="b0ad2fae034bac74a30d27e20446ccce538e15fb563926dabcfa3ea2812da9a3" Jul 17 17:54:34.731404 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.731387 2573 scope.go:117] "RemoveContainer" containerID="5ada1029c61544330fd5e6c6bb478d6f3314fcaee34cfa91cb475de78ab926f0" Jul 17 17:54:34.739337 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.739320 2573 scope.go:117] "RemoveContainer" containerID="b0ad2fae034bac74a30d27e20446ccce538e15fb563926dabcfa3ea2812da9a3" Jul 17 17:54:34.739582 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:54:34.739565 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ad2fae034bac74a30d27e20446ccce538e15fb563926dabcfa3ea2812da9a3\": container with ID starting with b0ad2fae034bac74a30d27e20446ccce538e15fb563926dabcfa3ea2812da9a3 not found: ID does not exist" containerID="b0ad2fae034bac74a30d27e20446ccce538e15fb563926dabcfa3ea2812da9a3" Jul 17 17:54:34.740121 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.739592 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ad2fae034bac74a30d27e20446ccce538e15fb563926dabcfa3ea2812da9a3"} err="failed to get container status \"b0ad2fae034bac74a30d27e20446ccce538e15fb563926dabcfa3ea2812da9a3\": rpc error: code = NotFound desc = could not find container \"b0ad2fae034bac74a30d27e20446ccce538e15fb563926dabcfa3ea2812da9a3\": container with ID starting with b0ad2fae034bac74a30d27e20446ccce538e15fb563926dabcfa3ea2812da9a3 not found: ID does not exist" Jul 17 17:54:34.740121 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.739608 2573 scope.go:117] "RemoveContainer" containerID="5ada1029c61544330fd5e6c6bb478d6f3314fcaee34cfa91cb475de78ab926f0" Jul 17 17:54:34.740121 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:54:34.739890 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ada1029c61544330fd5e6c6bb478d6f3314fcaee34cfa91cb475de78ab926f0\": container with ID starting with 5ada1029c61544330fd5e6c6bb478d6f3314fcaee34cfa91cb475de78ab926f0 not found: ID does not exist" containerID="5ada1029c61544330fd5e6c6bb478d6f3314fcaee34cfa91cb475de78ab926f0" Jul 17 17:54:34.740121 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.739915 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ada1029c61544330fd5e6c6bb478d6f3314fcaee34cfa91cb475de78ab926f0"} err="failed to get container status \"5ada1029c61544330fd5e6c6bb478d6f3314fcaee34cfa91cb475de78ab926f0\": rpc error: code = NotFound desc = could not find container \"5ada1029c61544330fd5e6c6bb478d6f3314fcaee34cfa91cb475de78ab926f0\": container with ID starting with 5ada1029c61544330fd5e6c6bb478d6f3314fcaee34cfa91cb475de78ab926f0 not found: ID does not exist" Jul 17 17:54:34.744831 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.744809 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh"] Jul 17 17:54:34.750989 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:34.750969 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ce07281e-openshift-default-68b9f68565-v5rfh"] Jul 17 17:54:35.007495 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.007463 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx"] Jul 17 17:54:35.038807 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.038761 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:35.135841 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.135795 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:35.141198 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.141167 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:54:35.269388 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.269304 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-openshift-service-ca\") pod \"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce\" (UID: \"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce\") " Jul 17 17:54:35.269388 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.269376 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-maas-api-tls\") pod \"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce\" (UID: \"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce\") " Jul 17 17:54:35.269610 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.269470 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkbbn\" (UniqueName: \"kubernetes.io/projected/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-kube-api-access-tkbbn\") pod \"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce\" (UID: \"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce\") " Jul 17 17:54:35.269708 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.269662 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" (UID: "2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:54:35.269828 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.269811 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:54:35.271537 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.271512 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" (UID: "2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:54:35.271637 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.271555 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-kube-api-access-tkbbn" (OuterVolumeSpecName: "kube-api-access-tkbbn") pod "2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" (UID: "2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce"). InnerVolumeSpecName "kube-api-access-tkbbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:54:35.370674 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.370641 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:54:35.370674 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.370672 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkbbn\" (UniqueName: \"kubernetes.io/projected/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce-kube-api-access-tkbbn\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:54:35.729494 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.729416 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" Jul 17 17:54:35.729494 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.729443 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx" event={"ID":"2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce","Type":"ContainerDied","Data":"cec5abf2a8bc07845a92643a0ccea5ccfc1d5a9af5f724260a048a537b41d428"} Jul 17 17:54:35.729699 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.729518 2573 scope.go:117] "RemoveContainer" containerID="f6a0a63a4322c314bfb6f583c4c4d8f41cb1861e7dc974f40468bd0dcf1259ed" Jul 17 17:54:35.756244 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.756214 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx"] Jul 17 17:54:35.763722 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.763700 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce07281e-55644d4946-mr4bx"] Jul 17 17:54:35.778277 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.778252 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" path="/var/lib/kubelet/pods/2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce/volumes" Jul 17 17:54:35.778735 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:35.778720 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bd185d4-27ae-4987-9176-e59138a35c46" path="/var/lib/kubelet/pods/8bd185d4-27ae-4987-9176-e59138a35c46/volumes" Jul 17 17:54:39.812153 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:39.812122 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:54:39.815168 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:39.815144 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:54:40.932210 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:40.932182 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:41.031975 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.031945 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:41.227073 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.226990 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:41.319042 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.319016 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:41.388260 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.388224 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd"] Jul 17 17:54:41.388877 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.388859 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:41.388877 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.388878 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:41.389030 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.388890 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd185d4-27ae-4987-9176-e59138a35c46" containerName="istio-proxy" Jul 17 17:54:41.389030 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.388899 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd185d4-27ae-4987-9176-e59138a35c46" containerName="istio-proxy" Jul 17 17:54:41.389030 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.388914 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:41.389030 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.388923 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:41.389030 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.388934 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd185d4-27ae-4987-9176-e59138a35c46" containerName="istio-proxy" Jul 17 17:54:41.389030 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.388942 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd185d4-27ae-4987-9176-e59138a35c46" containerName="istio-proxy" Jul 17 17:54:41.389030 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.388952 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:41.389030 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.388959 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:41.389030 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.388981 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:41.389030 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.388989 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:41.389030 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.388998 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:41.389030 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.389006 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:41.389386 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.389089 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:41.389386 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.389103 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:41.389386 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.389113 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:41.389386 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.389120 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:41.389386 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.389130 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd185d4-27ae-4987-9176-e59138a35c46" containerName="istio-proxy" Jul 17 17:54:41.389386 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.389139 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:41.392256 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.392234 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.396408 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.396387 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-36649ebd-gw-openshift-default-dockercfg-jrv52\"" Jul 17 17:54:41.405971 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.405946 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd"] Jul 17 17:54:41.425593 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.425568 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istiod-ca-cert\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.425716 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.425601 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-token\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.425716 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.425657 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-workload-socket\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.425716 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.425674 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-data\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.425716 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.425698 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-podinfo\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.425955 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.425800 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf8bp\" (UniqueName: \"kubernetes.io/projected/172a6d71-23a6-43f3-98e8-cdbc98522ec8-kube-api-access-rf8bp\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.425955 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.425846 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-credential-socket\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.425955 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.425906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-workload-certs\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.425955 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.425931 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-envoy\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.430666 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.430641 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:41.526889 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.526855 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-workload-socket\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.526889 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.526889 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-data\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.527133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.526914 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-podinfo\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.527133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.526954 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf8bp\" (UniqueName: \"kubernetes.io/projected/172a6d71-23a6-43f3-98e8-cdbc98522ec8-kube-api-access-rf8bp\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.527133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.526975 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-credential-socket\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.527133 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.527001 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-workload-certs\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.527334 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.527144 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-envoy\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.527334 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.527222 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istiod-ca-cert\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.527334 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.527255 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-token\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.527473 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.527342 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-data\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.527473 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.527384 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-workload-socket\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.527737 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.527716 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-credential-socket\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.527821 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.527737 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-workload-certs\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.528078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.528058 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istiod-ca-cert\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.529428 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.529407 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-envoy\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.529593 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.529578 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-podinfo\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.538578 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.538555 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-token\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.538958 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.538941 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf8bp\" (UniqueName: \"kubernetes.io/projected/172a6d71-23a6-43f3-98e8-cdbc98522ec8-kube-api-access-rf8bp\") pod \"e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.703015 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.702967 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:41.842124 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.842099 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd"] Jul 17 17:54:41.844216 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:54:41.844186 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172a6d71_23a6_43f3_98e8_cdbc98522ec8.slice/crio-b467c7ac47d7ff286d0cb96d1f3073deaf4f9128f53eb1c32ff1e2b288edafb1 WatchSource:0}: Error finding container b467c7ac47d7ff286d0cb96d1f3073deaf4f9128f53eb1c32ff1e2b288edafb1: Status 404 returned error can't find the container with id b467c7ac47d7ff286d0cb96d1f3073deaf4f9128f53eb1c32ff1e2b288edafb1 Jul 17 17:54:41.846272 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.846239 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:54:41.846372 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.846300 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:54:41.846372 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:41.846331 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:54:42.715709 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:42.715678 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:42.766079 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:42.766046 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" event={"ID":"172a6d71-23a6-43f3-98e8-cdbc98522ec8","Type":"ContainerStarted","Data":"f54c7d83b2a59d0f6468ac12a86227e121b226e9a72c6cbaa54c42f65690376e"} Jul 17 17:54:42.766262 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:42.766106 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" event={"ID":"172a6d71-23a6-43f3-98e8-cdbc98522ec8","Type":"ContainerStarted","Data":"b467c7ac47d7ff286d0cb96d1f3073deaf4f9128f53eb1c32ff1e2b288edafb1"} Jul 17 17:54:42.790137 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:42.790078 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" podStartSLOduration=1.790063125 podStartE2EDuration="1.790063125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:54:42.787491107 +0000 UTC m=+1803.522915153" watchObservedRunningTime="2026-07-17 17:54:42.790063125 +0000 UTC m=+1803.525487173" Jul 17 17:54:42.855464 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:42.855431 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr"] Jul 17 17:54:42.856159 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:42.856137 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:42.856274 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:42.856163 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:42.856332 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:42.856279 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd185d4-27ae-4987-9176-e59138a35c46" containerName="istio-proxy" Jul 17 17:54:42.856332 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:42.856293 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a9b2850-f0ff-4ad2-a5eb-115eb8d0f5ce" containerName="maas-api" Jul 17 17:54:42.859763 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:42.859742 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:54:42.862458 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:42.862436 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-36649ebd-serving-cert\"" Jul 17 17:54:42.869227 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:42.869204 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr"] Jul 17 17:54:42.888687 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:42.888657 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:42.943100 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:42.943068 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw2wf\" (UniqueName: \"kubernetes.io/projected/c0b71a86-d695-4e76-98bb-f729d9759083-kube-api-access-mw2wf\") pod \"maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr\" (UID: \"c0b71a86-d695-4e76-98bb-f729d9759083\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:54:42.943231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:42.943144 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b71a86-d695-4e76-98bb-f729d9759083-maas-api-tls\") pod \"maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr\" (UID: \"c0b71a86-d695-4e76-98bb-f729d9759083\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:54:42.943231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:42.943168 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b71a86-d695-4e76-98bb-f729d9759083-openshift-service-ca\") pod \"maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr\" (UID: \"c0b71a86-d695-4e76-98bb-f729d9759083\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:54:43.038554 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.038518 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:43.043797 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.043775 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw2wf\" (UniqueName: \"kubernetes.io/projected/c0b71a86-d695-4e76-98bb-f729d9759083-kube-api-access-mw2wf\") pod \"maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr\" (UID: \"c0b71a86-d695-4e76-98bb-f729d9759083\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:54:43.043899 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.043855 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b71a86-d695-4e76-98bb-f729d9759083-maas-api-tls\") pod \"maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr\" (UID: \"c0b71a86-d695-4e76-98bb-f729d9759083\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:54:43.043899 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.043880 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b71a86-d695-4e76-98bb-f729d9759083-openshift-service-ca\") pod \"maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr\" (UID: \"c0b71a86-d695-4e76-98bb-f729d9759083\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:54:43.044455 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.044437 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b71a86-d695-4e76-98bb-f729d9759083-openshift-service-ca\") pod \"maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr\" (UID: \"c0b71a86-d695-4e76-98bb-f729d9759083\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:54:43.046321 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.046304 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b71a86-d695-4e76-98bb-f729d9759083-maas-api-tls\") pod \"maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr\" (UID: \"c0b71a86-d695-4e76-98bb-f729d9759083\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:54:43.052114 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.052092 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw2wf\" (UniqueName: \"kubernetes.io/projected/c0b71a86-d695-4e76-98bb-f729d9759083-kube-api-access-mw2wf\") pod \"maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr\" (UID: \"c0b71a86-d695-4e76-98bb-f729d9759083\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:54:43.172195 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.172159 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:54:43.311055 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.311028 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr"] Jul 17 17:54:43.313019 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:54:43.312990 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b71a86_d695_4e76_98bb_f729d9759083.slice/crio-8c0bd1e450db69d690fac5d462938fa1a219da867fa30af80719ed622321eea1 WatchSource:0}: Error finding container 8c0bd1e450db69d690fac5d462938fa1a219da867fa30af80719ed622321eea1: Status 404 returned error can't find the container with id 8c0bd1e450db69d690fac5d462938fa1a219da867fa30af80719ed622321eea1 Jul 17 17:54:43.429078 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.429046 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:43.531021 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.530990 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:43.703358 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.703331 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:43.708453 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.708428 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:43.777182 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.777152 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" event={"ID":"c0b71a86-d695-4e76-98bb-f729d9759083","Type":"ContainerStarted","Data":"cb7ea933403a066ae5457bb82f75006dde2c8fd106e3755cfdfa12c9563435a6"} Jul 17 17:54:43.777499 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.777205 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:43.777499 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.777223 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" event={"ID":"c0b71a86-d695-4e76-98bb-f729d9759083","Type":"ContainerStarted","Data":"8c0bd1e450db69d690fac5d462938fa1a219da867fa30af80719ed622321eea1"} Jul 17 17:54:43.777499 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.777240 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:54:43.777499 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.777274 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:43.790946 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:43.790898 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" podStartSLOduration=1.434978892 podStartE2EDuration="1.790882711s" totalImagesPullingTime="355.903819ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:54:43.789213731 +0000 UTC m=+1804.524637781" watchObservedRunningTime="2026-07-17 17:54:43.790882711 +0000 UTC m=+1804.526306794" Jul 17 17:54:44.778692 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:44.778655 2573 generic.go:356] "Generic (PLEG): container finished" podID="c0b71a86-d695-4e76-98bb-f729d9759083" containerID="cb7ea933403a066ae5457bb82f75006dde2c8fd106e3755cfdfa12c9563435a6" exitCode=1 Jul 17 17:54:44.779062 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:44.778745 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" event={"ID":"c0b71a86-d695-4e76-98bb-f729d9759083","Type":"ContainerDied","Data":"cb7ea933403a066ae5457bb82f75006dde2c8fd106e3755cfdfa12c9563435a6"} Jul 17 17:54:44.779062 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:44.778911 2573 scope.go:117] "RemoveContainer" containerID="cb7ea933403a066ae5457bb82f75006dde2c8fd106e3755cfdfa12c9563435a6" Jul 17 17:54:45.784354 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:45.784325 2573 generic.go:356] "Generic (PLEG): container finished" podID="c0b71a86-d695-4e76-98bb-f729d9759083" containerID="7587744f5693479944a9ff022ddb0943272814e87f3ecc75d7a0c727ec59b48f" exitCode=1 Jul 17 17:54:45.784858 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:45.784406 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" event={"ID":"c0b71a86-d695-4e76-98bb-f729d9759083","Type":"ContainerDied","Data":"7587744f5693479944a9ff022ddb0943272814e87f3ecc75d7a0c727ec59b48f"} Jul 17 17:54:45.784858 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:45.784463 2573 scope.go:117] "RemoveContainer" containerID="cb7ea933403a066ae5457bb82f75006dde2c8fd106e3755cfdfa12c9563435a6" Jul 17 17:54:45.784858 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:45.784715 2573 scope.go:117] "RemoveContainer" containerID="7587744f5693479944a9ff022ddb0943272814e87f3ecc75d7a0c727ec59b48f" Jul 17 17:54:45.785007 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:54:45.784921 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr_odh-ai-gateway-infra(c0b71a86-d695-4e76-98bb-f729d9759083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" Jul 17 17:54:46.790672 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:46.790644 2573 scope.go:117] "RemoveContainer" containerID="7587744f5693479944a9ff022ddb0943272814e87f3ecc75d7a0c727ec59b48f" Jul 17 17:54:46.791019 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:54:46.790836 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr_odh-ai-gateway-infra(c0b71a86-d695-4e76-98bb-f729d9759083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" Jul 17 17:54:48.772951 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:48.772919 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:54:48.773340 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:48.773325 2573 scope.go:117] "RemoveContainer" containerID="7587744f5693479944a9ff022ddb0943272814e87f3ecc75d7a0c727ec59b48f" Jul 17 17:54:48.773525 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:54:48.773506 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr_odh-ai-gateway-infra(c0b71a86-d695-4e76-98bb-f729d9759083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" Jul 17 17:54:50.244600 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:50.244560 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:50.347970 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:50.347938 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:53.172408 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:53.172370 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:54:53.172956 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:53.172935 2573 scope.go:117] "RemoveContainer" containerID="7587744f5693479944a9ff022ddb0943272814e87f3ecc75d7a0c727ec59b48f" Jul 17 17:54:53.173167 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:54:53.173144 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr_odh-ai-gateway-infra(c0b71a86-d695-4e76-98bb-f729d9759083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" Jul 17 17:54:53.578986 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:53.578949 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:54.846998 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:54.846964 2573 generic.go:356] "Generic (PLEG): container finished" podID="172a6d71-23a6-43f3-98e8-cdbc98522ec8" containerID="f54c7d83b2a59d0f6468ac12a86227e121b226e9a72c6cbaa54c42f65690376e" exitCode=0 Jul 17 17:54:54.847425 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:54.847004 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" event={"ID":"172a6d71-23a6-43f3-98e8-cdbc98522ec8","Type":"ContainerDied","Data":"f54c7d83b2a59d0f6468ac12a86227e121b226e9a72c6cbaa54c42f65690376e"} Jul 17 17:54:54.847425 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:54.847376 2573 scope.go:117] "RemoveContainer" containerID="f54c7d83b2a59d0f6468ac12a86227e121b226e9a72c6cbaa54c42f65690376e" Jul 17 17:54:54.848346 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:54.848316 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:54:54.848440 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:54.848398 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:54:54.848500 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:54.848470 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:54:55.854231 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:55.854193 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" event={"ID":"172a6d71-23a6-43f3-98e8-cdbc98522ec8","Type":"ContainerStarted","Data":"f8a683f475355497bda41795526834f171fa95dc9ce83180b81304452430486f"} Jul 17 17:54:56.704754 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:56.704712 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:56.704966 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:56.704851 2573 patch_prober.go:28] interesting pod/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 17 17:54:56.704966 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:56.704910 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" podUID="172a6d71-23a6-43f3-98e8-cdbc98522ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 17 17:54:57.703443 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:57.703399 2573 patch_prober.go:28] interesting pod/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 17 17:54:57.704058 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:57.704021 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" podUID="172a6d71-23a6-43f3-98e8-cdbc98522ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 17 17:54:58.703686 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:58.703648 2573 patch_prober.go:28] interesting pod/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 17 17:54:58.704270 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:58.704237 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" podUID="172a6d71-23a6-43f3-98e8-cdbc98522ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 17 17:54:58.773174 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:58.773137 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:54:59.703735 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:59.703701 2573 patch_prober.go:28] interesting pod/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 17 17:54:59.704291 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:54:59.704256 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" podUID="172a6d71-23a6-43f3-98e8-cdbc98522ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 17 17:55:00.707561 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:00.707527 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:55:00.708549 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:00.708529 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:55:06.772123 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:06.772091 2573 scope.go:117] "RemoveContainer" containerID="7587744f5693479944a9ff022ddb0943272814e87f3ecc75d7a0c727ec59b48f" Jul 17 17:55:07.916094 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:07.916062 2573 generic.go:356] "Generic (PLEG): container finished" podID="c0b71a86-d695-4e76-98bb-f729d9759083" containerID="906941605e96a6d3ef8529167051626056b50d572c45b5d8d4849cd8e701642d" exitCode=1 Jul 17 17:55:07.916492 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:07.916136 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" event={"ID":"c0b71a86-d695-4e76-98bb-f729d9759083","Type":"ContainerDied","Data":"906941605e96a6d3ef8529167051626056b50d572c45b5d8d4849cd8e701642d"} Jul 17 17:55:07.916492 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:07.916202 2573 scope.go:117] "RemoveContainer" containerID="7587744f5693479944a9ff022ddb0943272814e87f3ecc75d7a0c727ec59b48f" Jul 17 17:55:07.916595 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:07.916547 2573 scope.go:117] "RemoveContainer" containerID="906941605e96a6d3ef8529167051626056b50d572c45b5d8d4849cd8e701642d" Jul 17 17:55:07.916789 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:55:07.916766 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr_odh-ai-gateway-infra(c0b71a86-d695-4e76-98bb-f729d9759083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" Jul 17 17:55:08.456364 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:08.456334 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:55:08.773658 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:08.773605 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:55:08.922539 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:08.922513 2573 scope.go:117] "RemoveContainer" containerID="906941605e96a6d3ef8529167051626056b50d572c45b5d8d4849cd8e701642d" Jul 17 17:55:08.922913 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:55:08.922739 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr_odh-ai-gateway-infra(c0b71a86-d695-4e76-98bb-f729d9759083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" Jul 17 17:55:13.172690 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:13.172646 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:55:13.173161 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:13.173133 2573 scope.go:117] "RemoveContainer" containerID="906941605e96a6d3ef8529167051626056b50d572c45b5d8d4849cd8e701642d" Jul 17 17:55:13.173363 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:55:13.173338 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr_odh-ai-gateway-infra(c0b71a86-d695-4e76-98bb-f729d9759083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" Jul 17 17:55:25.771854 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:25.771825 2573 scope.go:117] "RemoveContainer" containerID="906941605e96a6d3ef8529167051626056b50d572c45b5d8d4849cd8e701642d" Jul 17 17:55:25.772428 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:55:25.772092 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr_odh-ai-gateway-infra(c0b71a86-d695-4e76-98bb-f729d9759083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" Jul 17 17:55:38.771882 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:38.771851 2573 scope.go:117] "RemoveContainer" containerID="906941605e96a6d3ef8529167051626056b50d572c45b5d8d4849cd8e701642d" Jul 17 17:55:40.079782 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:40.079747 2573 generic.go:356] "Generic (PLEG): container finished" podID="c0b71a86-d695-4e76-98bb-f729d9759083" containerID="2c146865119ab78ba3f51dfb247912057826c701108485e465f480a53166a197" exitCode=1 Jul 17 17:55:40.080199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:40.079819 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" event={"ID":"c0b71a86-d695-4e76-98bb-f729d9759083","Type":"ContainerDied","Data":"2c146865119ab78ba3f51dfb247912057826c701108485e465f480a53166a197"} Jul 17 17:55:40.080199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:40.079907 2573 scope.go:117] "RemoveContainer" containerID="906941605e96a6d3ef8529167051626056b50d572c45b5d8d4849cd8e701642d" Jul 17 17:55:40.080365 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:40.080297 2573 scope.go:117] "RemoveContainer" containerID="2c146865119ab78ba3f51dfb247912057826c701108485e465f480a53166a197" Jul 17 17:55:40.080508 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:55:40.080490 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr_odh-ai-gateway-infra(c0b71a86-d695-4e76-98bb-f729d9759083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" Jul 17 17:55:43.172476 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:43.172444 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:55:43.172884 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:43.172834 2573 scope.go:117] "RemoveContainer" containerID="2c146865119ab78ba3f51dfb247912057826c701108485e465f480a53166a197" Jul 17 17:55:43.173020 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:55:43.173001 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr_odh-ai-gateway-infra(c0b71a86-d695-4e76-98bb-f729d9759083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" Jul 17 17:55:43.777940 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:43.777908 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:55:44.099940 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:44.099860 2573 scope.go:117] "RemoveContainer" containerID="2c146865119ab78ba3f51dfb247912057826c701108485e465f480a53166a197" Jul 17 17:55:44.100088 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:55:44.100055 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr_odh-ai-gateway-infra(c0b71a86-d695-4e76-98bb-f729d9759083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" Jul 17 17:55:47.457512 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:47.457464 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd"] Jul 17 17:55:47.458009 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:47.457744 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" podUID="172a6d71-23a6-43f3-98e8-cdbc98522ec8" containerName="istio-proxy" containerID="cri-o://f8a683f475355497bda41795526834f171fa95dc9ce83180b81304452430486f" gracePeriod=30 Jul 17 17:55:47.472311 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:47.472285 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:55:47.567213 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:47.567178 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:55:52.708004 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.707978 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:55:52.732551 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.732488 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-envoy\") pod \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " Jul 17 17:55:52.732551 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.732518 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istiod-ca-cert\") pod \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " Jul 17 17:55:52.732551 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.732539 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf8bp\" (UniqueName: \"kubernetes.io/projected/172a6d71-23a6-43f3-98e8-cdbc98522ec8-kube-api-access-rf8bp\") pod \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " Jul 17 17:55:52.732817 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.732556 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-token\") pod \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " Jul 17 17:55:52.732817 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.732588 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-workload-certs\") pod \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " Jul 17 17:55:52.732817 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.732657 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-data\") pod \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " Jul 17 17:55:52.732817 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.732701 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-workload-socket\") pod \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " Jul 17 17:55:52.732817 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.732737 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-credential-socket\") pod \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " Jul 17 17:55:52.732817 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.732802 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-podinfo\") pod \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\" (UID: \"172a6d71-23a6-43f3-98e8-cdbc98522ec8\") " Jul 17 17:55:52.733165 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.732872 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "172a6d71-23a6-43f3-98e8-cdbc98522ec8" (UID: "172a6d71-23a6-43f3-98e8-cdbc98522ec8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:55:52.733165 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.733094 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "172a6d71-23a6-43f3-98e8-cdbc98522ec8" (UID: "172a6d71-23a6-43f3-98e8-cdbc98522ec8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:55:52.733165 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.733112 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:55:52.733165 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.733114 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "172a6d71-23a6-43f3-98e8-cdbc98522ec8" (UID: "172a6d71-23a6-43f3-98e8-cdbc98522ec8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:55:52.735397 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.733167 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "172a6d71-23a6-43f3-98e8-cdbc98522ec8" (UID: "172a6d71-23a6-43f3-98e8-cdbc98522ec8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:55:52.735397 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.735073 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-data" (OuterVolumeSpecName: "istio-data") pod "172a6d71-23a6-43f3-98e8-cdbc98522ec8" (UID: "172a6d71-23a6-43f3-98e8-cdbc98522ec8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:55:52.735603 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.735571 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172a6d71-23a6-43f3-98e8-cdbc98522ec8-kube-api-access-rf8bp" (OuterVolumeSpecName: "kube-api-access-rf8bp") pod "172a6d71-23a6-43f3-98e8-cdbc98522ec8" (UID: "172a6d71-23a6-43f3-98e8-cdbc98522ec8"). InnerVolumeSpecName "kube-api-access-rf8bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:55:52.735967 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.735930 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-token" (OuterVolumeSpecName: "istio-token") pod "172a6d71-23a6-43f3-98e8-cdbc98522ec8" (UID: "172a6d71-23a6-43f3-98e8-cdbc98522ec8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:55:52.735967 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.735960 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "172a6d71-23a6-43f3-98e8-cdbc98522ec8" (UID: "172a6d71-23a6-43f3-98e8-cdbc98522ec8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:55:52.736244 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.736213 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "172a6d71-23a6-43f3-98e8-cdbc98522ec8" (UID: "172a6d71-23a6-43f3-98e8-cdbc98522ec8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 17:55:52.834602 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.834572 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:55:52.834602 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.834599 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:55:52.834769 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.834609 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:55:52.834769 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.834644 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:55:52.834769 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.834657 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:55:52.834769 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.834670 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf8bp\" (UniqueName: \"kubernetes.io/projected/172a6d71-23a6-43f3-98e8-cdbc98522ec8-kube-api-access-rf8bp\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:55:52.834769 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.834681 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:55:52.834769 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:52.834688 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172a6d71-23a6-43f3-98e8-cdbc98522ec8-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:55:53.143793 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.143763 2573 generic.go:356] "Generic (PLEG): container finished" podID="172a6d71-23a6-43f3-98e8-cdbc98522ec8" containerID="f8a683f475355497bda41795526834f171fa95dc9ce83180b81304452430486f" exitCode=0 Jul 17 17:55:53.143955 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.143815 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" Jul 17 17:55:53.143955 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.143852 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" event={"ID":"172a6d71-23a6-43f3-98e8-cdbc98522ec8","Type":"ContainerDied","Data":"f8a683f475355497bda41795526834f171fa95dc9ce83180b81304452430486f"} Jul 17 17:55:53.143955 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.143907 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd" event={"ID":"172a6d71-23a6-43f3-98e8-cdbc98522ec8","Type":"ContainerDied","Data":"b467c7ac47d7ff286d0cb96d1f3073deaf4f9128f53eb1c32ff1e2b288edafb1"} Jul 17 17:55:53.143955 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.143932 2573 scope.go:117] "RemoveContainer" containerID="f8a683f475355497bda41795526834f171fa95dc9ce83180b81304452430486f" Jul 17 17:55:53.152952 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.152937 2573 scope.go:117] "RemoveContainer" containerID="f54c7d83b2a59d0f6468ac12a86227e121b226e9a72c6cbaa54c42f65690376e" Jul 17 17:55:53.161399 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.161383 2573 scope.go:117] "RemoveContainer" containerID="f8a683f475355497bda41795526834f171fa95dc9ce83180b81304452430486f" Jul 17 17:55:53.161657 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:55:53.161636 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a683f475355497bda41795526834f171fa95dc9ce83180b81304452430486f\": container with ID starting with f8a683f475355497bda41795526834f171fa95dc9ce83180b81304452430486f not found: ID does not exist" containerID="f8a683f475355497bda41795526834f171fa95dc9ce83180b81304452430486f" Jul 17 17:55:53.161787 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.161670 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a683f475355497bda41795526834f171fa95dc9ce83180b81304452430486f"} err="failed to get container status \"f8a683f475355497bda41795526834f171fa95dc9ce83180b81304452430486f\": rpc error: code = NotFound desc = could not find container \"f8a683f475355497bda41795526834f171fa95dc9ce83180b81304452430486f\": container with ID starting with f8a683f475355497bda41795526834f171fa95dc9ce83180b81304452430486f not found: ID does not exist" Jul 17 17:55:53.161787 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.161693 2573 scope.go:117] "RemoveContainer" containerID="f54c7d83b2a59d0f6468ac12a86227e121b226e9a72c6cbaa54c42f65690376e" Jul 17 17:55:53.161933 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:55:53.161916 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f54c7d83b2a59d0f6468ac12a86227e121b226e9a72c6cbaa54c42f65690376e\": container with ID starting with f54c7d83b2a59d0f6468ac12a86227e121b226e9a72c6cbaa54c42f65690376e not found: ID does not exist" containerID="f54c7d83b2a59d0f6468ac12a86227e121b226e9a72c6cbaa54c42f65690376e" Jul 17 17:55:53.161973 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.161940 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f54c7d83b2a59d0f6468ac12a86227e121b226e9a72c6cbaa54c42f65690376e"} err="failed to get container status \"f54c7d83b2a59d0f6468ac12a86227e121b226e9a72c6cbaa54c42f65690376e\": rpc error: code = NotFound desc = could not find container \"f54c7d83b2a59d0f6468ac12a86227e121b226e9a72c6cbaa54c42f65690376e\": container with ID starting with f54c7d83b2a59d0f6468ac12a86227e121b226e9a72c6cbaa54c42f65690376e not found: ID does not exist" Jul 17 17:55:53.169891 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.169868 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd"] Jul 17 17:55:53.178055 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.178013 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-36649ebd-gw-openshift-default-568c568b5f-9rnsd"] Jul 17 17:55:53.407543 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.407465 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr"] Jul 17 17:55:53.432433 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.432404 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:55:53.542507 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.542478 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:55:53.643850 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.643823 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b71a86-d695-4e76-98bb-f729d9759083-maas-api-tls\") pod \"c0b71a86-d695-4e76-98bb-f729d9759083\" (UID: \"c0b71a86-d695-4e76-98bb-f729d9759083\") " Jul 17 17:55:53.644008 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.643879 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw2wf\" (UniqueName: \"kubernetes.io/projected/c0b71a86-d695-4e76-98bb-f729d9759083-kube-api-access-mw2wf\") pod \"c0b71a86-d695-4e76-98bb-f729d9759083\" (UID: \"c0b71a86-d695-4e76-98bb-f729d9759083\") " Jul 17 17:55:53.644008 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.643983 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b71a86-d695-4e76-98bb-f729d9759083-openshift-service-ca\") pod \"c0b71a86-d695-4e76-98bb-f729d9759083\" (UID: \"c0b71a86-d695-4e76-98bb-f729d9759083\") " Jul 17 17:55:53.644375 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.644349 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b71a86-d695-4e76-98bb-f729d9759083-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c0b71a86-d695-4e76-98bb-f729d9759083" (UID: "c0b71a86-d695-4e76-98bb-f729d9759083"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:55:53.645876 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.645858 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0b71a86-d695-4e76-98bb-f729d9759083-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c0b71a86-d695-4e76-98bb-f729d9759083" (UID: "c0b71a86-d695-4e76-98bb-f729d9759083"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:55:53.645953 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.645883 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b71a86-d695-4e76-98bb-f729d9759083-kube-api-access-mw2wf" (OuterVolumeSpecName: "kube-api-access-mw2wf") pod "c0b71a86-d695-4e76-98bb-f729d9759083" (UID: "c0b71a86-d695-4e76-98bb-f729d9759083"). InnerVolumeSpecName "kube-api-access-mw2wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:55:53.745683 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.745648 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b71a86-d695-4e76-98bb-f729d9759083-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:55:53.745683 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.745680 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b71a86-d695-4e76-98bb-f729d9759083-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:55:53.746071 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.745696 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw2wf\" (UniqueName: \"kubernetes.io/projected/c0b71a86-d695-4e76-98bb-f729d9759083-kube-api-access-mw2wf\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 17:55:53.752572 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.752548 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:55:53.778107 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.778064 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="172a6d71-23a6-43f3-98e8-cdbc98522ec8" path="/var/lib/kubelet/pods/172a6d71-23a6-43f3-98e8-cdbc98522ec8/volumes" Jul 17 17:55:53.854384 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:53.854350 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:55:54.151110 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:54.151033 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" Jul 17 17:55:54.151110 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:54.151075 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr" event={"ID":"c0b71a86-d695-4e76-98bb-f729d9759083","Type":"ContainerDied","Data":"8c0bd1e450db69d690fac5d462938fa1a219da867fa30af80719ed622321eea1"} Jul 17 17:55:54.151285 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:54.151135 2573 scope.go:117] "RemoveContainer" containerID="2c146865119ab78ba3f51dfb247912057826c701108485e465f480a53166a197" Jul 17 17:55:54.171452 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:54.171423 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr"] Jul 17 17:55:54.179963 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:54.179938 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-36649ebd-65fd4895c-fxfxr"] Jul 17 17:55:55.777997 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:55.777960 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" path="/var/lib/kubelet/pods/c0b71a86-d695-4e76-98bb-f729d9759083/volumes" Jul 17 17:55:59.416773 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.416732 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:55:59.518850 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.518811 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:55:59.625177 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.625140 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v"] Jul 17 17:55:59.625599 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.625580 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="172a6d71-23a6-43f3-98e8-cdbc98522ec8" containerName="istio-proxy" Jul 17 17:55:59.625719 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.625602 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="172a6d71-23a6-43f3-98e8-cdbc98522ec8" containerName="istio-proxy" Jul 17 17:55:59.625719 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.625645 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" containerName="maas-api" Jul 17 17:55:59.625719 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.625654 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" containerName="maas-api" Jul 17 17:55:59.625719 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.625671 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="172a6d71-23a6-43f3-98e8-cdbc98522ec8" containerName="istio-proxy" Jul 17 17:55:59.625719 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.625680 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="172a6d71-23a6-43f3-98e8-cdbc98522ec8" containerName="istio-proxy" Jul 17 17:55:59.625719 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.625693 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" containerName="maas-api" Jul 17 17:55:59.625719 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.625701 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" containerName="maas-api" Jul 17 17:55:59.625719 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.625710 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" containerName="maas-api" Jul 17 17:55:59.625719 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.625718 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" containerName="maas-api" Jul 17 17:55:59.626141 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.625826 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="172a6d71-23a6-43f3-98e8-cdbc98522ec8" containerName="istio-proxy" Jul 17 17:55:59.626141 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.625842 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="172a6d71-23a6-43f3-98e8-cdbc98522ec8" containerName="istio-proxy" Jul 17 17:55:59.626141 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.625852 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" containerName="maas-api" Jul 17 17:55:59.626141 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.625862 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" containerName="maas-api" Jul 17 17:55:59.626141 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.625873 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" containerName="maas-api" Jul 17 17:55:59.630568 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.630546 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.633058 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.633039 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-59516a0f-openshift-default-dockercfg-fz7bg\"" Jul 17 17:55:59.648926 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.648903 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v"] Jul 17 17:55:59.684706 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.679968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:55:59.706422 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.706385 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-token\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.706559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.706466 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-podinfo\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.706559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.706496 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjfp\" (UniqueName: \"kubernetes.io/projected/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-kube-api-access-6xjfp\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.706559 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.706534 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-workload-socket\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.706709 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.706557 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istiod-ca-cert\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.706709 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.706653 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-envoy\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.706709 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.706678 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-credential-socket\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.706828 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.706711 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-workload-certs\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.706828 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.706749 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-data\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.765383 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.765350 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:55:59.807521 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.807484 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-workload-certs\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.807731 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.807537 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-data\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.807731 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.807584 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-token\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.807731 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.807665 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-podinfo\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.807731 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.807695 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjfp\" (UniqueName: \"kubernetes.io/projected/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-kube-api-access-6xjfp\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.807964 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.807735 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-workload-socket\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.807964 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.807758 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istiod-ca-cert\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.807964 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.807815 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-envoy\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.807964 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.807843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-credential-socket\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.807964 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.807898 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-workload-certs\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.808227 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.807991 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-data\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.808227 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.808149 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-workload-socket\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.808227 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.808192 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-credential-socket\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.808496 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.808476 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istiod-ca-cert\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.810148 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.810120 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-envoy\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.810312 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.810295 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-podinfo\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.816227 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.816202 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-token\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.816341 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.816274 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjfp\" (UniqueName: \"kubernetes.io/projected/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-kube-api-access-6xjfp\") pod \"e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:55:59.942267 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:55:59.942185 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:56:00.079990 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.079966 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v"] Jul 17 17:56:00.082390 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:56:00.082348 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939bf544_4a9e_4d45_b47d_d3c8ebdfcc9a.slice/crio-beb7f321bef01026e919213ed69481b66d4b6f0348f2c0cc8e211eb6b5813f9e WatchSource:0}: Error finding container beb7f321bef01026e919213ed69481b66d4b6f0348f2c0cc8e211eb6b5813f9e: Status 404 returned error can't find the container with id beb7f321bef01026e919213ed69481b66d4b6f0348f2c0cc8e211eb6b5813f9e Jul 17 17:56:00.084565 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.084525 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:56:00.084658 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.084614 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:56:00.084716 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.084698 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:56:00.183744 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.183698 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" event={"ID":"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a","Type":"ContainerStarted","Data":"07da18b73bfb2ddfb44b337c1ba405530e97ac3e61be19b7b5f90edee8ebb93c"} Jul 17 17:56:00.183938 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.183762 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" event={"ID":"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a","Type":"ContainerStarted","Data":"beb7f321bef01026e919213ed69481b66d4b6f0348f2c0cc8e211eb6b5813f9e"} Jul 17 17:56:00.206741 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.206610 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" podStartSLOduration=1.206589048 podStartE2EDuration="1.206589048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:56:00.20317221 +0000 UTC m=+1880.938596285" watchObservedRunningTime="2026-07-17 17:56:00.206589048 +0000 UTC m=+1880.942013098" Jul 17 17:56:00.872751 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.872666 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:00.943227 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.943192 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:56:00.950150 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.950119 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:00.951175 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.951156 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24"] Jul 17 17:56:00.951553 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.951542 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" containerName="maas-api" Jul 17 17:56:00.951591 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.951554 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" containerName="maas-api" Jul 17 17:56:00.951658 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.951648 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b71a86-d695-4e76-98bb-f729d9759083" containerName="maas-api" Jul 17 17:56:00.955033 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.955016 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:56:00.957497 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.957484 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-59516a0f-serving-cert\"" Jul 17 17:56:00.966649 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:00.966608 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24"] Jul 17 17:56:01.018603 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:01.018567 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9f785a9-2a29-4b57-9855-3349daa41348-maas-api-tls\") pod \"maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24\" (UID: \"a9f785a9-2a29-4b57-9855-3349daa41348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:56:01.018764 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:01.018678 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f785a9-2a29-4b57-9855-3349daa41348-openshift-service-ca\") pod \"maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24\" (UID: \"a9f785a9-2a29-4b57-9855-3349daa41348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:56:01.018764 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:01.018759 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7k5p\" (UniqueName: \"kubernetes.io/projected/a9f785a9-2a29-4b57-9855-3349daa41348-kube-api-access-l7k5p\") pod \"maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24\" (UID: \"a9f785a9-2a29-4b57-9855-3349daa41348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:56:01.066485 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:01.066440 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:01.120291 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:01.120260 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7k5p\" (UniqueName: \"kubernetes.io/projected/a9f785a9-2a29-4b57-9855-3349daa41348-kube-api-access-l7k5p\") pod \"maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24\" (UID: \"a9f785a9-2a29-4b57-9855-3349daa41348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:56:01.120443 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:01.120344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9f785a9-2a29-4b57-9855-3349daa41348-maas-api-tls\") pod \"maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24\" (UID: \"a9f785a9-2a29-4b57-9855-3349daa41348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:56:01.120443 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:01.120378 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f785a9-2a29-4b57-9855-3349daa41348-openshift-service-ca\") pod \"maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24\" (UID: \"a9f785a9-2a29-4b57-9855-3349daa41348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:56:01.120515 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:56:01.120481 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-serving-cert: secret "maas-api-e2e-ait-59516a0f-serving-cert" not found Jul 17 17:56:01.120553 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:56:01.120540 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f785a9-2a29-4b57-9855-3349daa41348-maas-api-tls podName:a9f785a9-2a29-4b57-9855-3349daa41348 nodeName:}" failed. No retries permitted until 2026-07-17 17:56:01.620522836 +0000 UTC m=+1882.355946863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a9f785a9-2a29-4b57-9855-3349daa41348-maas-api-tls") pod "maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" (UID: "a9f785a9-2a29-4b57-9855-3349daa41348") : secret "maas-api-e2e-ait-59516a0f-serving-cert" not found Jul 17 17:56:01.121056 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:01.121035 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f785a9-2a29-4b57-9855-3349daa41348-openshift-service-ca\") pod \"maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24\" (UID: \"a9f785a9-2a29-4b57-9855-3349daa41348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:56:01.136602 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:01.136537 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7k5p\" (UniqueName: \"kubernetes.io/projected/a9f785a9-2a29-4b57-9855-3349daa41348-kube-api-access-l7k5p\") pod \"maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24\" (UID: \"a9f785a9-2a29-4b57-9855-3349daa41348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:56:01.291444 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:01.291413 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:01.624442 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:01.624401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9f785a9-2a29-4b57-9855-3349daa41348-maas-api-tls\") pod \"maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24\" (UID: \"a9f785a9-2a29-4b57-9855-3349daa41348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:56:01.626914 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:01.626894 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9f785a9-2a29-4b57-9855-3349daa41348-maas-api-tls\") pod \"maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24\" (UID: \"a9f785a9-2a29-4b57-9855-3349daa41348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:56:01.629233 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:01.629205 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:01.731523 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:01.731488 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:01.866728 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:01.866691 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:56:01.948741 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:01.948715 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:56:02.005853 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:02.005821 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24"] Jul 17 17:56:02.011125 ip-10-0-132-183 kubenswrapper[2573]: W0717 17:56:02.011096 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f785a9_2a29_4b57_9855_3349daa41348.slice/crio-9c44acc4b991d7c0e26887d6f79c7f41605e474b038a0c5ba50cbbcded845395 WatchSource:0}: Error finding container 9c44acc4b991d7c0e26887d6f79c7f41605e474b038a0c5ba50cbbcded845395: Status 404 returned error can't find the container with id 9c44acc4b991d7c0e26887d6f79c7f41605e474b038a0c5ba50cbbcded845395 Jul 17 17:56:02.195446 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:02.195349 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" event={"ID":"a9f785a9-2a29-4b57-9855-3349daa41348","Type":"ContainerStarted","Data":"9c44acc4b991d7c0e26887d6f79c7f41605e474b038a0c5ba50cbbcded845395"} Jul 17 17:56:02.195648 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:02.195607 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:56:02.196518 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:02.196499 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:56:02.980575 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:02.980546 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:03.201865 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:03.201771 2573 generic.go:356] "Generic (PLEG): container finished" podID="a9f785a9-2a29-4b57-9855-3349daa41348" containerID="3e117d0d7e34b3390325a80aaa5f7b1ccd1fd08a9a2cb62d4d7995b260e4a36b" exitCode=1 Jul 17 17:56:03.202002 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:03.201853 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" event={"ID":"a9f785a9-2a29-4b57-9855-3349daa41348","Type":"ContainerDied","Data":"3e117d0d7e34b3390325a80aaa5f7b1ccd1fd08a9a2cb62d4d7995b260e4a36b"} Jul 17 17:56:03.202170 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:03.202155 2573 scope.go:117] "RemoveContainer" containerID="3e117d0d7e34b3390325a80aaa5f7b1ccd1fd08a9a2cb62d4d7995b260e4a36b" Jul 17 17:56:04.208926 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:04.208827 2573 generic.go:356] "Generic (PLEG): container finished" podID="a9f785a9-2a29-4b57-9855-3349daa41348" containerID="fefcd75a6c0f0f56b4b162e1d7bb3a15688bfad1b40edacaf71fda608dc07d14" exitCode=1 Jul 17 17:56:04.209362 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:04.208916 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" event={"ID":"a9f785a9-2a29-4b57-9855-3349daa41348","Type":"ContainerDied","Data":"fefcd75a6c0f0f56b4b162e1d7bb3a15688bfad1b40edacaf71fda608dc07d14"} Jul 17 17:56:04.209362 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:04.209008 2573 scope.go:117] "RemoveContainer" containerID="3e117d0d7e34b3390325a80aaa5f7b1ccd1fd08a9a2cb62d4d7995b260e4a36b" Jul 17 17:56:04.209362 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:04.209228 2573 scope.go:117] "RemoveContainer" containerID="fefcd75a6c0f0f56b4b162e1d7bb3a15688bfad1b40edacaf71fda608dc07d14" Jul 17 17:56:04.209524 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:56:04.209457 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:56:05.219246 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:05.219216 2573 scope.go:117] "RemoveContainer" containerID="fefcd75a6c0f0f56b4b162e1d7bb3a15688bfad1b40edacaf71fda608dc07d14" Jul 17 17:56:05.219949 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:56:05.219563 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:56:06.867142 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:06.867107 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:56:06.867515 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:06.867500 2573 scope.go:117] "RemoveContainer" containerID="fefcd75a6c0f0f56b4b162e1d7bb3a15688bfad1b40edacaf71fda608dc07d14" Jul 17 17:56:06.867749 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:56:06.867729 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:56:11.867196 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:11.867108 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:56:11.867654 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:11.867545 2573 scope.go:117] "RemoveContainer" containerID="fefcd75a6c0f0f56b4b162e1d7bb3a15688bfad1b40edacaf71fda608dc07d14" Jul 17 17:56:11.867806 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:56:11.867760 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:56:12.260313 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:12.260280 2573 generic.go:356] "Generic (PLEG): container finished" podID="939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" containerID="07da18b73bfb2ddfb44b337c1ba405530e97ac3e61be19b7b5f90edee8ebb93c" exitCode=0 Jul 17 17:56:12.260481 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:12.260318 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" event={"ID":"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a","Type":"ContainerDied","Data":"07da18b73bfb2ddfb44b337c1ba405530e97ac3e61be19b7b5f90edee8ebb93c"} Jul 17 17:56:12.260737 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:12.260723 2573 scope.go:117] "RemoveContainer" containerID="07da18b73bfb2ddfb44b337c1ba405530e97ac3e61be19b7b5f90edee8ebb93c" Jul 17 17:56:12.261749 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:12.261719 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:56:12.261824 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:12.261791 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:56:12.261867 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:12.261822 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 17:56:13.267689 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:13.267653 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" event={"ID":"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a","Type":"ContainerStarted","Data":"f4b2cf1ddb5da92d6258b39da0d2303cb679ab7dcf1d0ae123146e7f746c477b"} Jul 17 17:56:13.942824 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:13.942753 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:56:13.943801 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:13.943778 2573 patch_prober.go:28] interesting pod/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 17 17:56:13.943904 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:13.943836 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" podUID="939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 17 17:56:14.942692 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:14.942646 2573 patch_prober.go:28] interesting pod/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 17 17:56:14.943169 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:14.942732 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" podUID="939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 17 17:56:15.943515 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:15.943477 2573 patch_prober.go:28] interesting pod/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 17 17:56:15.943938 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:15.943550 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" podUID="939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 17 17:56:16.943512 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:16.943471 2573 patch_prober.go:28] interesting pod/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 17 17:56:16.943751 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:16.943537 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" podUID="939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 17 17:56:17.196474 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:17.196386 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:56:17.946430 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:17.946398 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:56:17.947358 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:17.947333 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 17:56:21.970188 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:21.970152 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:25.771886 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:25.771855 2573 scope.go:117] "RemoveContainer" containerID="fefcd75a6c0f0f56b4b162e1d7bb3a15688bfad1b40edacaf71fda608dc07d14" Jul 17 17:56:25.772269 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:25.772179 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 17:56:26.333963 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:26.333933 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" event={"ID":"a9f785a9-2a29-4b57-9855-3349daa41348","Type":"ContainerStarted","Data":"2c08ea07666af5c9fe58e2eb3510e21bc3f851ffc51b4e9e3bb33264b6169a57"} Jul 17 17:56:26.334159 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:26.334129 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:56:26.353354 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:26.353306 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podStartSLOduration=25.075946551 podStartE2EDuration="26.353291143s" totalImagesPullingTime="1.277344592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:56:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:56:26.351294014 +0000 UTC m=+1907.086718064" watchObservedRunningTime="2026-07-17 17:56:26.353291143 +0000 UTC m=+1907.088715191" Jul 17 17:56:27.340393 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:27.340360 2573 generic.go:356] "Generic (PLEG): container finished" podID="a9f785a9-2a29-4b57-9855-3349daa41348" containerID="2c08ea07666af5c9fe58e2eb3510e21bc3f851ffc51b4e9e3bb33264b6169a57" exitCode=1 Jul 17 17:56:27.340842 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:27.340451 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" event={"ID":"a9f785a9-2a29-4b57-9855-3349daa41348","Type":"ContainerDied","Data":"2c08ea07666af5c9fe58e2eb3510e21bc3f851ffc51b4e9e3bb33264b6169a57"} Jul 17 17:56:27.340842 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:27.340553 2573 scope.go:117] "RemoveContainer" containerID="fefcd75a6c0f0f56b4b162e1d7bb3a15688bfad1b40edacaf71fda608dc07d14" Jul 17 17:56:27.340842 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:27.340672 2573 scope.go:117] "RemoveContainer" containerID="2c08ea07666af5c9fe58e2eb3510e21bc3f851ffc51b4e9e3bb33264b6169a57" Jul 17 17:56:27.340993 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:56:27.340911 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:56:28.347151 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:28.347126 2573 scope.go:117] "RemoveContainer" containerID="2c08ea07666af5c9fe58e2eb3510e21bc3f851ffc51b4e9e3bb33264b6169a57" Jul 17 17:56:28.347509 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:56:28.347329 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:56:31.866986 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:31.866949 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:56:31.867433 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:31.867331 2573 scope.go:117] "RemoveContainer" containerID="2c08ea07666af5c9fe58e2eb3510e21bc3f851ffc51b4e9e3bb33264b6169a57" Jul 17 17:56:31.867522 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:56:31.867503 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:56:43.771972 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:43.771944 2573 scope.go:117] "RemoveContainer" containerID="2c08ea07666af5c9fe58e2eb3510e21bc3f851ffc51b4e9e3bb33264b6169a57" Jul 17 17:56:43.772369 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:56:43.772156 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:56:55.771845 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:55.771809 2573 scope.go:117] "RemoveContainer" containerID="2c08ea07666af5c9fe58e2eb3510e21bc3f851ffc51b4e9e3bb33264b6169a57" Jul 17 17:56:56.493574 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:56.493541 2573 generic.go:356] "Generic (PLEG): container finished" podID="a9f785a9-2a29-4b57-9855-3349daa41348" containerID="07219fd6d33392754908e87a06b751ccda8f4b1c4bc5cdd840d043f09131f025" exitCode=1 Jul 17 17:56:56.493709 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:56.493610 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" event={"ID":"a9f785a9-2a29-4b57-9855-3349daa41348","Type":"ContainerDied","Data":"07219fd6d33392754908e87a06b751ccda8f4b1c4bc5cdd840d043f09131f025"} Jul 17 17:56:56.493709 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:56.493688 2573 scope.go:117] "RemoveContainer" containerID="2c08ea07666af5c9fe58e2eb3510e21bc3f851ffc51b4e9e3bb33264b6169a57" Jul 17 17:56:56.494016 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:56:56.493991 2573 scope.go:117] "RemoveContainer" containerID="07219fd6d33392754908e87a06b751ccda8f4b1c4bc5cdd840d043f09131f025" Jul 17 17:56:56.494208 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:56:56.494186 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:57:01.334914 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:01.334866 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:57:01.335331 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:01.335302 2573 scope.go:117] "RemoveContainer" containerID="07219fd6d33392754908e87a06b751ccda8f4b1c4bc5cdd840d043f09131f025" Jul 17 17:57:01.335532 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:57:01.335511 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:57:01.867838 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:01.867806 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:57:01.868173 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:01.868158 2573 scope.go:117] "RemoveContainer" containerID="07219fd6d33392754908e87a06b751ccda8f4b1c4bc5cdd840d043f09131f025" Jul 17 17:57:01.868353 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:57:01.868333 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:57:09.785723 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:09.785687 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:09.883199 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:09.883165 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:15.771612 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:15.771575 2573 scope.go:117] "RemoveContainer" containerID="07219fd6d33392754908e87a06b751ccda8f4b1c4bc5cdd840d043f09131f025" Jul 17 17:57:15.772098 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:57:15.771849 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:57:28.772043 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:28.772015 2573 scope.go:117] "RemoveContainer" containerID="07219fd6d33392754908e87a06b751ccda8f4b1c4bc5cdd840d043f09131f025" Jul 17 17:57:28.772403 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:57:28.772194 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:57:43.771522 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:43.771493 2573 scope.go:117] "RemoveContainer" containerID="07219fd6d33392754908e87a06b751ccda8f4b1c4bc5cdd840d043f09131f025" Jul 17 17:57:44.737710 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:44.737679 2573 generic.go:356] "Generic (PLEG): container finished" podID="a9f785a9-2a29-4b57-9855-3349daa41348" containerID="c20a60a2e917f88d3f411525ca4c9c9f9361d03e0a18ec0dece8cca41add6534" exitCode=1 Jul 17 17:57:44.737878 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:44.737723 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" event={"ID":"a9f785a9-2a29-4b57-9855-3349daa41348","Type":"ContainerDied","Data":"c20a60a2e917f88d3f411525ca4c9c9f9361d03e0a18ec0dece8cca41add6534"} Jul 17 17:57:44.737878 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:44.737771 2573 scope.go:117] "RemoveContainer" containerID="07219fd6d33392754908e87a06b751ccda8f4b1c4bc5cdd840d043f09131f025" Jul 17 17:57:44.738096 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:44.738078 2573 scope.go:117] "RemoveContainer" containerID="c20a60a2e917f88d3f411525ca4c9c9f9361d03e0a18ec0dece8cca41add6534" Jul 17 17:57:44.738305 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:57:44.738284 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:57:46.334978 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:46.334930 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:57:46.335429 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:46.335329 2573 scope.go:117] "RemoveContainer" containerID="c20a60a2e917f88d3f411525ca4c9c9f9361d03e0a18ec0dece8cca41add6534" Jul 17 17:57:46.335527 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:57:46.335508 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:57:51.867644 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:51.867590 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:57:51.868137 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:57:51.868070 2573 scope.go:117] "RemoveContainer" containerID="c20a60a2e917f88d3f411525ca4c9c9f9361d03e0a18ec0dece8cca41add6534" Jul 17 17:57:51.868332 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:57:51.868307 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:58:06.771509 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:58:06.771478 2573 scope.go:117] "RemoveContainer" containerID="c20a60a2e917f88d3f411525ca4c9c9f9361d03e0a18ec0dece8cca41add6534" Jul 17 17:58:06.771899 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:58:06.771730 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:58:17.771795 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:58:17.771762 2573 scope.go:117] "RemoveContainer" containerID="c20a60a2e917f88d3f411525ca4c9c9f9361d03e0a18ec0dece8cca41add6534" Jul 17 17:58:17.772270 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:58:17.771910 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:58:31.772020 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:58:31.771983 2573 scope.go:117] "RemoveContainer" containerID="c20a60a2e917f88d3f411525ca4c9c9f9361d03e0a18ec0dece8cca41add6534" Jul 17 17:58:31.772410 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:58:31.772271 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:58:45.772235 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:58:45.772203 2573 scope.go:117] "RemoveContainer" containerID="c20a60a2e917f88d3f411525ca4c9c9f9361d03e0a18ec0dece8cca41add6534" Jul 17 17:58:45.772765 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:58:45.772380 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:58:59.774794 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:58:59.774765 2573 scope.go:117] "RemoveContainer" containerID="c20a60a2e917f88d3f411525ca4c9c9f9361d03e0a18ec0dece8cca41add6534" Jul 17 17:58:59.775298 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:58:59.774970 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:59:11.771756 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:11.771673 2573 scope.go:117] "RemoveContainer" containerID="c20a60a2e917f88d3f411525ca4c9c9f9361d03e0a18ec0dece8cca41add6534" Jul 17 17:59:13.176257 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:13.176221 2573 generic.go:356] "Generic (PLEG): container finished" podID="a9f785a9-2a29-4b57-9855-3349daa41348" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" exitCode=1 Jul 17 17:59:13.176672 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:13.176290 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" event={"ID":"a9f785a9-2a29-4b57-9855-3349daa41348","Type":"ContainerDied","Data":"821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679"} Jul 17 17:59:13.176672 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:13.176348 2573 scope.go:117] "RemoveContainer" containerID="c20a60a2e917f88d3f411525ca4c9c9f9361d03e0a18ec0dece8cca41add6534" Jul 17 17:59:13.176672 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:13.176661 2573 scope.go:117] "RemoveContainer" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" Jul 17 17:59:13.176897 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:59:13.176871 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:59:16.335255 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:16.335222 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:59:16.335694 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:16.335612 2573 scope.go:117] "RemoveContainer" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" Jul 17 17:59:16.335845 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:59:16.335823 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:59:21.867706 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:21.867670 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 17:59:21.868071 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:21.868008 2573 scope.go:117] "RemoveContainer" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" Jul 17 17:59:21.868206 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:59:21.868185 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:59:24.699846 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:24.699802 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:24.787027 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:24.786990 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:27.756354 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:27.756312 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:28.930432 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:28.930393 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:33.771724 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:33.771691 2573 scope.go:117] "RemoveContainer" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" Jul 17 17:59:33.772117 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:59:33.771944 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:59:39.844892 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:39.844866 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:59:39.849093 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:39.849074 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 17:59:46.771472 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:46.771435 2573 scope.go:117] "RemoveContainer" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" Jul 17 17:59:46.771874 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:59:46.771712 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 17:59:59.773940 ip-10-0-132-183 kubenswrapper[2573]: I0717 17:59:59.773908 2573 scope.go:117] "RemoveContainer" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" Jul 17 17:59:59.774391 ip-10-0-132-183 kubenswrapper[2573]: E0717 17:59:59.774146 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 18:00:11.771994 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:00:11.771962 2573 scope.go:117] "RemoveContainer" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" Jul 17 18:00:11.772599 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:00:11.772149 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 18:00:24.771849 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:00:24.771814 2573 scope.go:117] "RemoveContainer" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" Jul 17 18:00:24.772257 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:00:24.772004 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 18:00:36.771853 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:00:36.771822 2573 scope.go:117] "RemoveContainer" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" Jul 17 18:00:36.772250 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:00:36.771981 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 18:00:51.771860 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:00:51.771823 2573 scope.go:117] "RemoveContainer" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" Jul 17 18:00:51.772360 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:00:51.772026 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 18:01:05.772208 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:01:05.772172 2573 scope.go:117] "RemoveContainer" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" Jul 17 18:01:05.772680 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:01:05.772405 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 18:01:19.774451 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:01:19.774421 2573 scope.go:117] "RemoveContainer" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" Jul 17 18:01:19.776945 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:01:19.774607 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 18:01:32.771817 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:01:32.771788 2573 scope.go:117] "RemoveContainer" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" Jul 17 18:01:32.772228 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:01:32.771980 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 18:01:43.771699 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:01:43.771670 2573 scope.go:117] "RemoveContainer" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" Jul 17 18:01:43.772043 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:01:43.771881 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 18:01:56.771810 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:01:56.771779 2573 scope.go:117] "RemoveContainer" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" Jul 17 18:01:56.772187 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:01:56.772085 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:01:59.008102 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:01:59.008065 2573 generic.go:356] "Generic (PLEG): container finished" podID="a9f785a9-2a29-4b57-9855-3349daa41348" containerID="45b28d8484b1f5c1d365740128d2145696b29b26f6b37ddc9c3bff029132d440" exitCode=1 Jul 17 18:01:59.008493 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:01:59.008106 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" event={"ID":"a9f785a9-2a29-4b57-9855-3349daa41348","Type":"ContainerDied","Data":"45b28d8484b1f5c1d365740128d2145696b29b26f6b37ddc9c3bff029132d440"} Jul 17 18:01:59.008493 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:01:59.008179 2573 scope.go:117] "RemoveContainer" containerID="821777e93f64622db43d3a83dcbd349dd3caef02fc52141881b9afc6234cc679" Jul 17 18:01:59.008640 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:01:59.008531 2573 scope.go:117] "RemoveContainer" containerID="45b28d8484b1f5c1d365740128d2145696b29b26f6b37ddc9c3bff029132d440" Jul 17 18:01:59.008784 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:01:59.008760 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 18:02:01.334297 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:01.334219 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 18:02:01.334677 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:01.334576 2573 scope.go:117] "RemoveContainer" containerID="45b28d8484b1f5c1d365740128d2145696b29b26f6b37ddc9c3bff029132d440" Jul 17 18:02:01.334842 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:02:01.334818 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 18:02:01.867223 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:01.867187 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 18:02:02.026585 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:02.026556 2573 scope.go:117] "RemoveContainer" containerID="45b28d8484b1f5c1d365740128d2145696b29b26f6b37ddc9c3bff029132d440" Jul 17 18:02:02.026807 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:02:02.026786 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24_odh-ai-gateway-infra(a9f785a9-2a29-4b57-9855-3349daa41348)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" Jul 17 18:02:06.114409 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:06.114365 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v"] Jul 17 18:02:06.115034 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:06.114644 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" podUID="939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" containerName="istio-proxy" containerID="cri-o://f4b2cf1ddb5da92d6258b39da0d2303cb679ab7dcf1d0ae123146e7f746c477b" gracePeriod=30 Jul 17 18:02:06.382508 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:06.382400 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:06.455055 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:06.455023 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:11.375417 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.375347 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 18:02:11.415246 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.415210 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-podinfo\") pod \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " Jul 17 18:02:11.415430 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.415277 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xjfp\" (UniqueName: \"kubernetes.io/projected/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-kube-api-access-6xjfp\") pod \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " Jul 17 18:02:11.415430 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.415359 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-data\") pod \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " Jul 17 18:02:11.415430 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.415409 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-workload-certs\") pod \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " Jul 17 18:02:11.415634 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.415444 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-token\") pod \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " Jul 17 18:02:11.415634 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.415504 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istiod-ca-cert\") pod \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " Jul 17 18:02:11.415634 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.415539 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-workload-socket\") pod \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " Jul 17 18:02:11.415844 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.415814 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" (UID: "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:02:11.416172 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.416142 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-envoy\") pod \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " Jul 17 18:02:11.416279 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.416209 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-credential-socket\") pod \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\" (UID: \"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a\") " Jul 17 18:02:11.416341 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.416144 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" (UID: "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:02:11.416341 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.416297 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-data" (OuterVolumeSpecName: "istio-data") pod "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" (UID: "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:02:11.417405 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.416662 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:02:11.417405 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.416698 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:02:11.417405 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.416711 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:02:11.417405 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.416793 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" (UID: "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:02:11.417405 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.416839 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" (UID: "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:02:11.418910 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.418881 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-kube-api-access-6xjfp" (OuterVolumeSpecName: "kube-api-access-6xjfp") pod "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" (UID: "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a"). InnerVolumeSpecName "kube-api-access-6xjfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:02:11.419027 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.418958 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" (UID: "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:02:11.419781 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.419750 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" (UID: "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 18:02:11.420105 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.420047 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-token" (OuterVolumeSpecName: "istio-token") pod "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" (UID: "939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:02:11.422675 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.422366 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24"] Jul 17 18:02:11.517307 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.517271 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:02:11.517307 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.517306 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xjfp\" (UniqueName: \"kubernetes.io/projected/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-kube-api-access-6xjfp\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:02:11.517509 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.517321 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:02:11.517509 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.517332 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:02:11.517509 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.517343 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:02:11.517509 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.517354 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:02:11.555907 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.555812 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:11.564503 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.564475 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 18:02:11.618453 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.618414 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9f785a9-2a29-4b57-9855-3349daa41348-maas-api-tls\") pod \"a9f785a9-2a29-4b57-9855-3349daa41348\" (UID: \"a9f785a9-2a29-4b57-9855-3349daa41348\") " Jul 17 18:02:11.618662 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.618555 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f785a9-2a29-4b57-9855-3349daa41348-openshift-service-ca\") pod \"a9f785a9-2a29-4b57-9855-3349daa41348\" (UID: \"a9f785a9-2a29-4b57-9855-3349daa41348\") " Jul 17 18:02:11.618662 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.618585 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7k5p\" (UniqueName: \"kubernetes.io/projected/a9f785a9-2a29-4b57-9855-3349daa41348-kube-api-access-l7k5p\") pod \"a9f785a9-2a29-4b57-9855-3349daa41348\" (UID: \"a9f785a9-2a29-4b57-9855-3349daa41348\") " Jul 17 18:02:11.618939 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.618914 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f785a9-2a29-4b57-9855-3349daa41348-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a9f785a9-2a29-4b57-9855-3349daa41348" (UID: "a9f785a9-2a29-4b57-9855-3349daa41348"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:02:11.620531 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.620506 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f785a9-2a29-4b57-9855-3349daa41348-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a9f785a9-2a29-4b57-9855-3349daa41348" (UID: "a9f785a9-2a29-4b57-9855-3349daa41348"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:02:11.620657 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.620532 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f785a9-2a29-4b57-9855-3349daa41348-kube-api-access-l7k5p" (OuterVolumeSpecName: "kube-api-access-l7k5p") pod "a9f785a9-2a29-4b57-9855-3349daa41348" (UID: "a9f785a9-2a29-4b57-9855-3349daa41348"). InnerVolumeSpecName "kube-api-access-l7k5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:02:11.719809 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.719715 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f785a9-2a29-4b57-9855-3349daa41348-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:02:11.719809 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.719747 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7k5p\" (UniqueName: \"kubernetes.io/projected/a9f785a9-2a29-4b57-9855-3349daa41348-kube-api-access-l7k5p\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:02:11.719809 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:11.719757 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9f785a9-2a29-4b57-9855-3349daa41348-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:02:12.079009 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.078972 2573 generic.go:356] "Generic (PLEG): container finished" podID="939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" containerID="f4b2cf1ddb5da92d6258b39da0d2303cb679ab7dcf1d0ae123146e7f746c477b" exitCode=0 Jul 17 18:02:12.079170 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.079029 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" Jul 17 18:02:12.079170 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.079058 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" event={"ID":"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a","Type":"ContainerDied","Data":"f4b2cf1ddb5da92d6258b39da0d2303cb679ab7dcf1d0ae123146e7f746c477b"} Jul 17 18:02:12.079170 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.079112 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v" event={"ID":"939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a","Type":"ContainerDied","Data":"beb7f321bef01026e919213ed69481b66d4b6f0348f2c0cc8e211eb6b5813f9e"} Jul 17 18:02:12.079170 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.079144 2573 scope.go:117] "RemoveContainer" containerID="f4b2cf1ddb5da92d6258b39da0d2303cb679ab7dcf1d0ae123146e7f746c477b" Jul 17 18:02:12.080500 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.080473 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" Jul 17 18:02:12.080637 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.080499 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24" event={"ID":"a9f785a9-2a29-4b57-9855-3349daa41348","Type":"ContainerDied","Data":"9c44acc4b991d7c0e26887d6f79c7f41605e474b038a0c5ba50cbbcded845395"} Jul 17 18:02:12.088261 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.088245 2573 scope.go:117] "RemoveContainer" containerID="07da18b73bfb2ddfb44b337c1ba405530e97ac3e61be19b7b5f90edee8ebb93c" Jul 17 18:02:12.096205 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.096189 2573 scope.go:117] "RemoveContainer" containerID="f4b2cf1ddb5da92d6258b39da0d2303cb679ab7dcf1d0ae123146e7f746c477b" Jul 17 18:02:12.096421 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:02:12.096401 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b2cf1ddb5da92d6258b39da0d2303cb679ab7dcf1d0ae123146e7f746c477b\": container with ID starting with f4b2cf1ddb5da92d6258b39da0d2303cb679ab7dcf1d0ae123146e7f746c477b not found: ID does not exist" containerID="f4b2cf1ddb5da92d6258b39da0d2303cb679ab7dcf1d0ae123146e7f746c477b" Jul 17 18:02:12.096476 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.096428 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b2cf1ddb5da92d6258b39da0d2303cb679ab7dcf1d0ae123146e7f746c477b"} err="failed to get container status \"f4b2cf1ddb5da92d6258b39da0d2303cb679ab7dcf1d0ae123146e7f746c477b\": rpc error: code = NotFound desc = could not find container \"f4b2cf1ddb5da92d6258b39da0d2303cb679ab7dcf1d0ae123146e7f746c477b\": container with ID starting with f4b2cf1ddb5da92d6258b39da0d2303cb679ab7dcf1d0ae123146e7f746c477b not found: ID does not exist" Jul 17 18:02:12.096476 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.096445 2573 scope.go:117] "RemoveContainer" containerID="07da18b73bfb2ddfb44b337c1ba405530e97ac3e61be19b7b5f90edee8ebb93c" Jul 17 18:02:12.096723 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:02:12.096699 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07da18b73bfb2ddfb44b337c1ba405530e97ac3e61be19b7b5f90edee8ebb93c\": container with ID starting with 07da18b73bfb2ddfb44b337c1ba405530e97ac3e61be19b7b5f90edee8ebb93c not found: ID does not exist" containerID="07da18b73bfb2ddfb44b337c1ba405530e97ac3e61be19b7b5f90edee8ebb93c" Jul 17 18:02:12.096802 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.096731 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07da18b73bfb2ddfb44b337c1ba405530e97ac3e61be19b7b5f90edee8ebb93c"} err="failed to get container status \"07da18b73bfb2ddfb44b337c1ba405530e97ac3e61be19b7b5f90edee8ebb93c\": rpc error: code = NotFound desc = could not find container \"07da18b73bfb2ddfb44b337c1ba405530e97ac3e61be19b7b5f90edee8ebb93c\": container with ID starting with 07da18b73bfb2ddfb44b337c1ba405530e97ac3e61be19b7b5f90edee8ebb93c not found: ID does not exist" Jul 17 18:02:12.096802 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.096754 2573 scope.go:117] "RemoveContainer" containerID="45b28d8484b1f5c1d365740128d2145696b29b26f6b37ddc9c3bff029132d440" Jul 17 18:02:12.099921 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.099902 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24"] Jul 17 18:02:12.106319 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.106303 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59516a0f-57598cd5bd-5gm24"] Jul 17 18:02:12.117505 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.117472 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v"] Jul 17 18:02:12.119240 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.119220 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-59516a0f-openshift-default-7b75cdcfb9-x4f8v"] Jul 17 18:02:12.445369 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:12.445286 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:13.777309 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:13.777274 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" path="/var/lib/kubelet/pods/939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a/volumes" Jul 17 18:02:13.777712 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:13.777697 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" path="/var/lib/kubelet/pods/a9f785a9-2a29-4b57-9855-3349daa41348/volumes" Jul 17 18:02:17.999961 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:17.999922 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:18.144635 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.144587 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:18.267087 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267004 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98"] Jul 17 18:02:18.267552 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267530 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:18.267552 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267552 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:18.267759 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267567 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:18.267759 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267575 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:18.267759 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267595 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:18.267759 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267603 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:18.267759 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267615 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:18.267759 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267644 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:18.267759 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267667 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" containerName="istio-proxy" Jul 17 18:02:18.267759 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267675 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" containerName="istio-proxy" Jul 17 18:02:18.268128 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267790 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:18.268128 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267802 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" containerName="istio-proxy" Jul 17 18:02:18.268128 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267816 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:18.268128 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267825 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" containerName="istio-proxy" Jul 17 18:02:18.268128 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267833 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:18.268128 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267842 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:18.268128 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.267850 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:18.272916 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.272897 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.276653 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.276607 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e49d4a9a-openshift-default-dockercfg-tdr5j\"" Jul 17 18:02:18.294771 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.294749 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98"] Jul 17 18:02:18.340081 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.340040 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:18.386990 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.386957 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-data\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.387172 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.386998 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-workload-socket\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.387172 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.387075 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-podinfo\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.387172 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.387124 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-envoy\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.387317 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.387173 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smxvq\" (UniqueName: \"kubernetes.io/projected/3a36cfcb-66b1-4928-8497-f600d59f02f0-kube-api-access-smxvq\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.387317 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.387229 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-credential-socket\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.387413 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.387332 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a36cfcb-66b1-4928-8497-f600d59f02f0-istiod-ca-cert\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.387413 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.387374 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-workload-certs\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.387474 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.387441 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-token\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.488299 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.488263 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smxvq\" (UniqueName: \"kubernetes.io/projected/3a36cfcb-66b1-4928-8497-f600d59f02f0-kube-api-access-smxvq\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.488511 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.488323 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-credential-socket\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.488511 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.488375 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a36cfcb-66b1-4928-8497-f600d59f02f0-istiod-ca-cert\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.488511 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.488398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-workload-certs\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.488511 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.488425 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-token\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.488511 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.488459 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-data\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.488511 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.488482 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-workload-socket\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.488511 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.488502 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-podinfo\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.488932 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.488534 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-envoy\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.488932 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.488873 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-credential-socket\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.489011 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.488962 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-workload-socket\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.489011 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.488994 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-workload-certs\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.489077 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.489026 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-data\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.489221 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.489195 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a36cfcb-66b1-4928-8497-f600d59f02f0-istiod-ca-cert\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.490967 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.490940 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-envoy\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.491172 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.491154 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-podinfo\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.497163 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.497140 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-token\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.497321 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.497301 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smxvq\" (UniqueName: \"kubernetes.io/projected/3a36cfcb-66b1-4928-8497-f600d59f02f0-kube-api-access-smxvq\") pod \"e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.583766 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.583683 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:18.720345 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.720310 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98"] Jul 17 18:02:18.723063 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:02:18.723037 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a36cfcb_66b1_4928_8497_f600d59f02f0.slice/crio-205e3d55e575cc580b619ab0025a9bc2e9f81548232a63744797bf8ca7669e83 WatchSource:0}: Error finding container 205e3d55e575cc580b619ab0025a9bc2e9f81548232a63744797bf8ca7669e83: Status 404 returned error can't find the container with id 205e3d55e575cc580b619ab0025a9bc2e9f81548232a63744797bf8ca7669e83 Jul 17 18:02:18.725179 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.725146 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:02:18.725260 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.725209 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:02:18.725260 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:18.725236 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:02:19.120370 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.120328 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" event={"ID":"3a36cfcb-66b1-4928-8497-f600d59f02f0","Type":"ContainerStarted","Data":"a23902a7d994ff1f325048c0005c080945543427ff466e9cd30c3cc331e31e21"} Jul 17 18:02:19.120826 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.120397 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" event={"ID":"3a36cfcb-66b1-4928-8497-f600d59f02f0","Type":"ContainerStarted","Data":"205e3d55e575cc580b619ab0025a9bc2e9f81548232a63744797bf8ca7669e83"} Jul 17 18:02:19.141130 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.141075 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" podStartSLOduration=1.14105947 podStartE2EDuration="1.14105947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:02:19.140752626 +0000 UTC m=+2259.876176696" watchObservedRunningTime="2026-07-17 18:02:19.14105947 +0000 UTC m=+2259.876483518" Jul 17 18:02:19.493262 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.493216 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:19.584874 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.584834 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:19.590081 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.590057 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:19.673545 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.673514 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2"] Jul 17 18:02:19.673963 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.673949 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" containerName="istio-proxy" Jul 17 18:02:19.674026 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.673967 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="939bf544-4a9e-4d45-b47d-d3c8ebdfcc9a" containerName="istio-proxy" Jul 17 18:02:19.674026 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.673992 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:19.674026 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.673997 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:19.674026 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.674008 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:19.674026 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.674014 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:19.674026 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.674021 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:19.674026 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.674026 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:19.674249 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.674100 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:19.674249 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.674109 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f785a9-2a29-4b57-9855-3349daa41348" containerName="maas-api" Jul 17 18:02:19.677204 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.677187 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:02:19.680246 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.680221 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e49d4a9a-serving-cert\"" Jul 17 18:02:19.697993 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.697966 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2"] Jul 17 18:02:19.764140 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.764037 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:19.805093 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.805061 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67845c4f-c0a5-464c-a9b2-43596544fab6-maas-api-tls\") pod \"maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2\" (UID: \"67845c4f-c0a5-464c-a9b2-43596544fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:02:19.805281 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.805117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67845c4f-c0a5-464c-a9b2-43596544fab6-openshift-service-ca\") pod \"maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2\" (UID: \"67845c4f-c0a5-464c-a9b2-43596544fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:02:19.805333 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.805283 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzwzs\" (UniqueName: \"kubernetes.io/projected/67845c4f-c0a5-464c-a9b2-43596544fab6-kube-api-access-bzwzs\") pod \"maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2\" (UID: \"67845c4f-c0a5-464c-a9b2-43596544fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:02:19.881957 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.881923 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:19.906682 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.906655 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzwzs\" (UniqueName: \"kubernetes.io/projected/67845c4f-c0a5-464c-a9b2-43596544fab6-kube-api-access-bzwzs\") pod \"maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2\" (UID: \"67845c4f-c0a5-464c-a9b2-43596544fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:02:19.906857 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.906705 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67845c4f-c0a5-464c-a9b2-43596544fab6-maas-api-tls\") pod \"maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2\" (UID: \"67845c4f-c0a5-464c-a9b2-43596544fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:02:19.906857 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.906740 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67845c4f-c0a5-464c-a9b2-43596544fab6-openshift-service-ca\") pod \"maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2\" (UID: \"67845c4f-c0a5-464c-a9b2-43596544fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:02:19.907348 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.907327 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67845c4f-c0a5-464c-a9b2-43596544fab6-openshift-service-ca\") pod \"maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2\" (UID: \"67845c4f-c0a5-464c-a9b2-43596544fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:02:19.909203 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.909179 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67845c4f-c0a5-464c-a9b2-43596544fab6-maas-api-tls\") pod \"maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2\" (UID: \"67845c4f-c0a5-464c-a9b2-43596544fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:02:19.917411 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.917380 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzwzs\" (UniqueName: \"kubernetes.io/projected/67845c4f-c0a5-464c-a9b2-43596544fab6-kube-api-access-bzwzs\") pod \"maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2\" (UID: \"67845c4f-c0a5-464c-a9b2-43596544fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:02:19.988399 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:19.988363 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:02:20.120131 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:20.120108 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2"] Jul 17 18:02:20.121816 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:02:20.121786 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67845c4f_c0a5_464c_a9b2_43596544fab6.slice/crio-8799fefc4b2f08d3d808b6466bb147787b9e82884979edeb9c77dd11cac226f7 WatchSource:0}: Error finding container 8799fefc4b2f08d3d808b6466bb147787b9e82884979edeb9c77dd11cac226f7: Status 404 returned error can't find the container with id 8799fefc4b2f08d3d808b6466bb147787b9e82884979edeb9c77dd11cac226f7 Jul 17 18:02:20.125230 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:20.125206 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:20.126317 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:20.126295 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:20.463148 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:20.463062 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:20.682909 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:20.682880 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:21.132388 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:21.132349 2573 generic.go:356] "Generic (PLEG): container finished" podID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerID="de28426572cff16475506633c4869d06e20258575469310d1a3439354c296173" exitCode=1 Jul 17 18:02:21.132794 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:21.132436 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" event={"ID":"67845c4f-c0a5-464c-a9b2-43596544fab6","Type":"ContainerDied","Data":"de28426572cff16475506633c4869d06e20258575469310d1a3439354c296173"} Jul 17 18:02:21.132794 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:21.132493 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" event={"ID":"67845c4f-c0a5-464c-a9b2-43596544fab6","Type":"ContainerStarted","Data":"8799fefc4b2f08d3d808b6466bb147787b9e82884979edeb9c77dd11cac226f7"} Jul 17 18:02:21.132794 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:21.132714 2573 scope.go:117] "RemoveContainer" containerID="de28426572cff16475506633c4869d06e20258575469310d1a3439354c296173" Jul 17 18:02:22.139605 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:22.139521 2573 generic.go:356] "Generic (PLEG): container finished" podID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerID="73d34366a300a50411a9ca129c5f3d870c1dd5f235d743256ff4fd842d7ee445" exitCode=1 Jul 17 18:02:22.140096 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:22.139603 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" event={"ID":"67845c4f-c0a5-464c-a9b2-43596544fab6","Type":"ContainerDied","Data":"73d34366a300a50411a9ca129c5f3d870c1dd5f235d743256ff4fd842d7ee445"} Jul 17 18:02:22.140096 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:22.139686 2573 scope.go:117] "RemoveContainer" containerID="de28426572cff16475506633c4869d06e20258575469310d1a3439354c296173" Jul 17 18:02:22.140096 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:22.139863 2573 scope.go:117] "RemoveContainer" containerID="73d34366a300a50411a9ca129c5f3d870c1dd5f235d743256ff4fd842d7ee445" Jul 17 18:02:22.140306 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:02:22.140103 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:02:23.147639 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:23.147603 2573 scope.go:117] "RemoveContainer" containerID="73d34366a300a50411a9ca129c5f3d870c1dd5f235d743256ff4fd842d7ee445" Jul 17 18:02:23.148019 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:02:23.147893 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:02:24.988884 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:24.988846 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:02:24.989369 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:24.989352 2573 scope.go:117] "RemoveContainer" containerID="73d34366a300a50411a9ca129c5f3d870c1dd5f235d743256ff4fd842d7ee445" Jul 17 18:02:24.989651 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:02:24.989604 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:02:29.989507 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:29.989467 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:02:29.991012 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:29.990029 2573 scope.go:117] "RemoveContainer" containerID="73d34366a300a50411a9ca129c5f3d870c1dd5f235d743256ff4fd842d7ee445" Jul 17 18:02:29.991012 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:02:29.990272 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:02:30.185427 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:30.185396 2573 generic.go:356] "Generic (PLEG): container finished" podID="3a36cfcb-66b1-4928-8497-f600d59f02f0" containerID="a23902a7d994ff1f325048c0005c080945543427ff466e9cd30c3cc331e31e21" exitCode=0 Jul 17 18:02:30.185588 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:30.185471 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" event={"ID":"3a36cfcb-66b1-4928-8497-f600d59f02f0","Type":"ContainerDied","Data":"a23902a7d994ff1f325048c0005c080945543427ff466e9cd30c3cc331e31e21"} Jul 17 18:02:30.185893 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:30.185879 2573 scope.go:117] "RemoveContainer" containerID="a23902a7d994ff1f325048c0005c080945543427ff466e9cd30c3cc331e31e21" Jul 17 18:02:30.186889 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:30.186862 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:02:30.186958 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:30.186930 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:02:30.186994 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:30.186962 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:02:31.192054 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:31.192018 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" event={"ID":"3a36cfcb-66b1-4928-8497-f600d59f02f0","Type":"ContainerStarted","Data":"821fdf9763ccdda676e8e50fdc5240e683a71541d569d7f743b79c02abf0a249"} Jul 17 18:02:31.584718 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:31.584679 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:31.585502 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:31.585476 2573 patch_prober.go:28] interesting pod/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 17 18:02:31.585677 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:31.585522 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" podUID="3a36cfcb-66b1-4928-8497-f600d59f02f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 17 18:02:32.584334 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:32.584299 2573 patch_prober.go:28] interesting pod/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 17 18:02:32.584732 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:32.584359 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" podUID="3a36cfcb-66b1-4928-8497-f600d59f02f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 17 18:02:33.584296 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:33.584260 2573 patch_prober.go:28] interesting pod/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 17 18:02:33.584478 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:33.584312 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" podUID="3a36cfcb-66b1-4928-8497-f600d59f02f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 17 18:02:34.584893 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:34.584856 2573 patch_prober.go:28] interesting pod/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 17 18:02:34.585293 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:34.584915 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" podUID="3a36cfcb-66b1-4928-8497-f600d59f02f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 17 18:02:35.125659 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:35.125609 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:35.588121 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:35.588092 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:35.589107 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:35.589088 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:02:37.691175 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:37.691142 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:37.786639 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:37.786591 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:40.771528 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:40.771496 2573 scope.go:117] "RemoveContainer" containerID="73d34366a300a50411a9ca129c5f3d870c1dd5f235d743256ff4fd842d7ee445" Jul 17 18:02:41.250079 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:41.250038 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" event={"ID":"67845c4f-c0a5-464c-a9b2-43596544fab6","Type":"ContainerStarted","Data":"bed0f9ef12a8d872ee3e83aab9c8aed7659ddf4030c52c84a7aa81538c4f28fa"} Jul 17 18:02:41.250305 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:41.250282 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:02:41.272705 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:41.272651 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podStartSLOduration=21.113876221 podStartE2EDuration="22.272635978s" totalImagesPullingTime="1.158759757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:02:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:02:41.270027905 +0000 UTC m=+2282.005451989" watchObservedRunningTime="2026-07-17 18:02:41.272635978 +0000 UTC m=+2282.008060018" Jul 17 18:02:41.540333 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:41.540305 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:42.257318 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:42.257283 2573 generic.go:356] "Generic (PLEG): container finished" podID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerID="bed0f9ef12a8d872ee3e83aab9c8aed7659ddf4030c52c84a7aa81538c4f28fa" exitCode=1 Jul 17 18:02:42.257767 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:42.257364 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" event={"ID":"67845c4f-c0a5-464c-a9b2-43596544fab6","Type":"ContainerDied","Data":"bed0f9ef12a8d872ee3e83aab9c8aed7659ddf4030c52c84a7aa81538c4f28fa"} Jul 17 18:02:42.257767 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:42.257433 2573 scope.go:117] "RemoveContainer" containerID="73d34366a300a50411a9ca129c5f3d870c1dd5f235d743256ff4fd842d7ee445" Jul 17 18:02:42.257767 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:42.257599 2573 scope.go:117] "RemoveContainer" containerID="bed0f9ef12a8d872ee3e83aab9c8aed7659ddf4030c52c84a7aa81538c4f28fa" Jul 17 18:02:42.257890 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:02:42.257849 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:02:43.264797 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:43.264770 2573 scope.go:117] "RemoveContainer" containerID="bed0f9ef12a8d872ee3e83aab9c8aed7659ddf4030c52c84a7aa81538c4f28fa" Jul 17 18:02:43.265191 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:02:43.264981 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:02:49.989238 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:49.989198 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:02:49.989675 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:02:49.989659 2573 scope.go:117] "RemoveContainer" containerID="bed0f9ef12a8d872ee3e83aab9c8aed7659ddf4030c52c84a7aa81538c4f28fa" Jul 17 18:02:49.989867 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:02:49.989849 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:03:01.771979 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:01.771943 2573 scope.go:117] "RemoveContainer" containerID="bed0f9ef12a8d872ee3e83aab9c8aed7659ddf4030c52c84a7aa81538c4f28fa" Jul 17 18:03:02.363012 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:02.362981 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" event={"ID":"67845c4f-c0a5-464c-a9b2-43596544fab6","Type":"ContainerStarted","Data":"29ef122f22b7f7b4b4f0ff9ea4a63cfa05c06b06ed8be56619f70ed2845e2d97"} Jul 17 18:03:02.363190 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:02.363176 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:03:03.369021 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:03.368990 2573 generic.go:356] "Generic (PLEG): container finished" podID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerID="29ef122f22b7f7b4b4f0ff9ea4a63cfa05c06b06ed8be56619f70ed2845e2d97" exitCode=1 Jul 17 18:03:03.369450 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:03.369064 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" event={"ID":"67845c4f-c0a5-464c-a9b2-43596544fab6","Type":"ContainerDied","Data":"29ef122f22b7f7b4b4f0ff9ea4a63cfa05c06b06ed8be56619f70ed2845e2d97"} Jul 17 18:03:03.369450 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:03.369126 2573 scope.go:117] "RemoveContainer" containerID="bed0f9ef12a8d872ee3e83aab9c8aed7659ddf4030c52c84a7aa81538c4f28fa" Jul 17 18:03:03.369548 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:03.369518 2573 scope.go:117] "RemoveContainer" containerID="29ef122f22b7f7b4b4f0ff9ea4a63cfa05c06b06ed8be56619f70ed2845e2d97" Jul 17 18:03:03.369781 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:03:03.369757 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:03:04.375495 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:04.375468 2573 scope.go:117] "RemoveContainer" containerID="29ef122f22b7f7b4b4f0ff9ea4a63cfa05c06b06ed8be56619f70ed2845e2d97" Jul 17 18:03:04.375887 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:03:04.375711 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:03:09.989422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:09.989372 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:03:09.989880 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:09.989859 2573 scope.go:117] "RemoveContainer" containerID="29ef122f22b7f7b4b4f0ff9ea4a63cfa05c06b06ed8be56619f70ed2845e2d97" Jul 17 18:03:09.990072 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:03:09.990050 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:03:22.771543 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:22.771509 2573 scope.go:117] "RemoveContainer" containerID="29ef122f22b7f7b4b4f0ff9ea4a63cfa05c06b06ed8be56619f70ed2845e2d97" Jul 17 18:03:22.772064 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:03:22.771739 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:03:36.771579 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:36.771548 2573 scope.go:117] "RemoveContainer" containerID="29ef122f22b7f7b4b4f0ff9ea4a63cfa05c06b06ed8be56619f70ed2845e2d97" Jul 17 18:03:36.772002 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:03:36.771794 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:03:49.774594 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:49.774558 2573 scope.go:117] "RemoveContainer" containerID="29ef122f22b7f7b4b4f0ff9ea4a63cfa05c06b06ed8be56619f70ed2845e2d97" Jul 17 18:03:50.611999 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:50.611969 2573 generic.go:356] "Generic (PLEG): container finished" podID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerID="a903b979d356404c58bfe2d3b33c84f34d897aa9f9bb3cfc374c0a88d37c321c" exitCode=1 Jul 17 18:03:50.612169 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:50.612048 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" event={"ID":"67845c4f-c0a5-464c-a9b2-43596544fab6","Type":"ContainerDied","Data":"a903b979d356404c58bfe2d3b33c84f34d897aa9f9bb3cfc374c0a88d37c321c"} Jul 17 18:03:50.612169 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:50.612116 2573 scope.go:117] "RemoveContainer" containerID="29ef122f22b7f7b4b4f0ff9ea4a63cfa05c06b06ed8be56619f70ed2845e2d97" Jul 17 18:03:50.612419 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:50.612403 2573 scope.go:117] "RemoveContainer" containerID="a903b979d356404c58bfe2d3b33c84f34d897aa9f9bb3cfc374c0a88d37c321c" Jul 17 18:03:50.612645 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:03:50.612608 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:03:52.363244 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:52.363210 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:03:52.363709 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:52.363550 2573 scope.go:117] "RemoveContainer" containerID="a903b979d356404c58bfe2d3b33c84f34d897aa9f9bb3cfc374c0a88d37c321c" Jul 17 18:03:52.363772 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:03:52.363752 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:03:59.988965 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:59.988930 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:03:59.989422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:03:59.989296 2573 scope.go:117] "RemoveContainer" containerID="a903b979d356404c58bfe2d3b33c84f34d897aa9f9bb3cfc374c0a88d37c321c" Jul 17 18:03:59.989510 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:03:59.989491 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:04:14.771398 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:04:14.771368 2573 scope.go:117] "RemoveContainer" containerID="a903b979d356404c58bfe2d3b33c84f34d897aa9f9bb3cfc374c0a88d37c321c" Jul 17 18:04:14.771798 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:04:14.771550 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:04:25.703466 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:04:25.703430 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:04:25.835382 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:04:25.835346 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:04:26.503064 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:04:26.503035 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:04:26.602685 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:04:26.602656 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:04:27.771948 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:04:27.771918 2573 scope.go:117] "RemoveContainer" containerID="a903b979d356404c58bfe2d3b33c84f34d897aa9f9bb3cfc374c0a88d37c321c" Jul 17 18:04:27.772347 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:04:27.772109 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:04:39.882585 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:04:39.882557 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 18:04:39.884161 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:04:39.884139 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 18:04:42.771518 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:04:42.771488 2573 scope.go:117] "RemoveContainer" containerID="a903b979d356404c58bfe2d3b33c84f34d897aa9f9bb3cfc374c0a88d37c321c" Jul 17 18:04:42.771913 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:04:42.771747 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:04:57.771382 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:04:57.771342 2573 scope.go:117] "RemoveContainer" containerID="a903b979d356404c58bfe2d3b33c84f34d897aa9f9bb3cfc374c0a88d37c321c" Jul 17 18:04:57.771875 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:04:57.771602 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:05:12.772308 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:12.772221 2573 scope.go:117] "RemoveContainer" containerID="a903b979d356404c58bfe2d3b33c84f34d897aa9f9bb3cfc374c0a88d37c321c" Jul 17 18:05:14.028684 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:14.028613 2573 generic.go:356] "Generic (PLEG): container finished" podID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerID="a8bb27cfc15b1d9e8f6880530ce074e6a9f4fc14113850f1e8d137b564991838" exitCode=1 Jul 17 18:05:14.028684 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:14.028666 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" event={"ID":"67845c4f-c0a5-464c-a9b2-43596544fab6","Type":"ContainerDied","Data":"a8bb27cfc15b1d9e8f6880530ce074e6a9f4fc14113850f1e8d137b564991838"} Jul 17 18:05:14.029256 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:14.028761 2573 scope.go:117] "RemoveContainer" containerID="a903b979d356404c58bfe2d3b33c84f34d897aa9f9bb3cfc374c0a88d37c321c" Jul 17 18:05:14.029317 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:14.029258 2573 scope.go:117] "RemoveContainer" containerID="a8bb27cfc15b1d9e8f6880530ce074e6a9f4fc14113850f1e8d137b564991838" Jul 17 18:05:14.029526 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:05:14.029500 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:05:17.363825 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:17.363785 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:05:17.364249 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:17.364232 2573 scope.go:117] "RemoveContainer" containerID="a8bb27cfc15b1d9e8f6880530ce074e6a9f4fc14113850f1e8d137b564991838" Jul 17 18:05:17.364448 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:05:17.364428 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:05:19.989057 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:19.989014 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:05:19.989694 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:19.989536 2573 scope.go:117] "RemoveContainer" containerID="a8bb27cfc15b1d9e8f6880530ce074e6a9f4fc14113850f1e8d137b564991838" Jul 17 18:05:19.989846 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:05:19.989821 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2_odh-ai-gateway-infra(67845c4f-c0a5-464c-a9b2-43596544fab6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" Jul 17 18:05:23.965985 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:23.965950 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98"] Jul 17 18:05:23.966388 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:23.966168 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" podUID="3a36cfcb-66b1-4928-8497-f600d59f02f0" containerName="istio-proxy" containerID="cri-o://821fdf9763ccdda676e8e50fdc5240e683a71541d569d7f743b79c02abf0a249" gracePeriod=30 Jul 17 18:05:24.006643 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:24.006594 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:25.263240 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:25.263201 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:29.105168 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.105126 2573 generic.go:356] "Generic (PLEG): container finished" podID="3a36cfcb-66b1-4928-8497-f600d59f02f0" containerID="821fdf9763ccdda676e8e50fdc5240e683a71541d569d7f743b79c02abf0a249" exitCode=0 Jul 17 18:05:29.105579 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.105194 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" event={"ID":"3a36cfcb-66b1-4928-8497-f600d59f02f0","Type":"ContainerDied","Data":"821fdf9763ccdda676e8e50fdc5240e683a71541d569d7f743b79c02abf0a249"} Jul 17 18:05:29.105579 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.105274 2573 scope.go:117] "RemoveContainer" containerID="a23902a7d994ff1f325048c0005c080945543427ff466e9cd30c3cc331e31e21" Jul 17 18:05:29.226833 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.226798 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:05:29.382866 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.382838 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a36cfcb-66b1-4928-8497-f600d59f02f0-istiod-ca-cert\") pod \"3a36cfcb-66b1-4928-8497-f600d59f02f0\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " Jul 17 18:05:29.383033 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.382882 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-workload-certs\") pod \"3a36cfcb-66b1-4928-8497-f600d59f02f0\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " Jul 17 18:05:29.383033 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.382931 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-envoy\") pod \"3a36cfcb-66b1-4928-8497-f600d59f02f0\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " Jul 17 18:05:29.383033 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.382991 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-credential-socket\") pod \"3a36cfcb-66b1-4928-8497-f600d59f02f0\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " Jul 17 18:05:29.383033 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.383021 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-podinfo\") pod \"3a36cfcb-66b1-4928-8497-f600d59f02f0\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " Jul 17 18:05:29.383218 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.383058 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smxvq\" (UniqueName: \"kubernetes.io/projected/3a36cfcb-66b1-4928-8497-f600d59f02f0-kube-api-access-smxvq\") pod \"3a36cfcb-66b1-4928-8497-f600d59f02f0\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " Jul 17 18:05:29.383218 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.383080 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-workload-socket\") pod \"3a36cfcb-66b1-4928-8497-f600d59f02f0\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " Jul 17 18:05:29.383218 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.383100 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-token\") pod \"3a36cfcb-66b1-4928-8497-f600d59f02f0\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " Jul 17 18:05:29.383218 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.383125 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-data\") pod \"3a36cfcb-66b1-4928-8497-f600d59f02f0\" (UID: \"3a36cfcb-66b1-4928-8497-f600d59f02f0\") " Jul 17 18:05:29.383413 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.383245 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a36cfcb-66b1-4928-8497-f600d59f02f0" (UID: "3a36cfcb-66b1-4928-8497-f600d59f02f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:05:29.383413 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.383274 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a36cfcb-66b1-4928-8497-f600d59f02f0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a36cfcb-66b1-4928-8497-f600d59f02f0" (UID: "3a36cfcb-66b1-4928-8497-f600d59f02f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:05:29.383413 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.383398 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a36cfcb-66b1-4928-8497-f600d59f02f0" (UID: "3a36cfcb-66b1-4928-8497-f600d59f02f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:05:29.383575 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.383541 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:05:29.383575 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.383560 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a36cfcb-66b1-4928-8497-f600d59f02f0-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:05:29.383575 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.383573 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:05:29.383780 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.383576 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a36cfcb-66b1-4928-8497-f600d59f02f0" (UID: "3a36cfcb-66b1-4928-8497-f600d59f02f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:05:29.384068 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.384046 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a36cfcb-66b1-4928-8497-f600d59f02f0" (UID: "3a36cfcb-66b1-4928-8497-f600d59f02f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:05:29.385320 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.385294 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a36cfcb-66b1-4928-8497-f600d59f02f0" (UID: "3a36cfcb-66b1-4928-8497-f600d59f02f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:05:29.385438 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.385389 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a36cfcb-66b1-4928-8497-f600d59f02f0-kube-api-access-smxvq" (OuterVolumeSpecName: "kube-api-access-smxvq") pod "3a36cfcb-66b1-4928-8497-f600d59f02f0" (UID: "3a36cfcb-66b1-4928-8497-f600d59f02f0"). InnerVolumeSpecName "kube-api-access-smxvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:05:29.385591 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.385575 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a36cfcb-66b1-4928-8497-f600d59f02f0" (UID: "3a36cfcb-66b1-4928-8497-f600d59f02f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:05:29.385717 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.385603 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a36cfcb-66b1-4928-8497-f600d59f02f0" (UID: "3a36cfcb-66b1-4928-8497-f600d59f02f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 18:05:29.485121 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.485070 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:05:29.485121 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.485117 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smxvq\" (UniqueName: \"kubernetes.io/projected/3a36cfcb-66b1-4928-8497-f600d59f02f0-kube-api-access-smxvq\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:05:29.485121 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.485129 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:05:29.485121 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.485138 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:05:29.485404 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.485147 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:05:29.485404 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:29.485155 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a36cfcb-66b1-4928-8497-f600d59f02f0-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:05:30.066192 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.066112 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2"] Jul 17 18:05:30.104842 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.104810 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:30.111121 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.111091 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" Jul 17 18:05:30.111505 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.111132 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98" event={"ID":"3a36cfcb-66b1-4928-8497-f600d59f02f0","Type":"ContainerDied","Data":"205e3d55e575cc580b619ab0025a9bc2e9f81548232a63744797bf8ca7669e83"} Jul 17 18:05:30.111505 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.111188 2573 scope.go:117] "RemoveContainer" containerID="821fdf9763ccdda676e8e50fdc5240e683a71541d569d7f743b79c02abf0a249" Jul 17 18:05:30.141718 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.141688 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98"] Jul 17 18:05:30.148168 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.148140 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e49d4a9a-openshift-default-66d89f68d7-nnc98"] Jul 17 18:05:30.208413 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.208387 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:05:30.292831 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.292796 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzwzs\" (UniqueName: \"kubernetes.io/projected/67845c4f-c0a5-464c-a9b2-43596544fab6-kube-api-access-bzwzs\") pod \"67845c4f-c0a5-464c-a9b2-43596544fab6\" (UID: \"67845c4f-c0a5-464c-a9b2-43596544fab6\") " Jul 17 18:05:30.293018 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.292852 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67845c4f-c0a5-464c-a9b2-43596544fab6-maas-api-tls\") pod \"67845c4f-c0a5-464c-a9b2-43596544fab6\" (UID: \"67845c4f-c0a5-464c-a9b2-43596544fab6\") " Jul 17 18:05:30.293018 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.292873 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67845c4f-c0a5-464c-a9b2-43596544fab6-openshift-service-ca\") pod \"67845c4f-c0a5-464c-a9b2-43596544fab6\" (UID: \"67845c4f-c0a5-464c-a9b2-43596544fab6\") " Jul 17 18:05:30.293291 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.293266 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67845c4f-c0a5-464c-a9b2-43596544fab6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67845c4f-c0a5-464c-a9b2-43596544fab6" (UID: "67845c4f-c0a5-464c-a9b2-43596544fab6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:05:30.294993 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.294971 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67845c4f-c0a5-464c-a9b2-43596544fab6-kube-api-access-bzwzs" (OuterVolumeSpecName: "kube-api-access-bzwzs") pod "67845c4f-c0a5-464c-a9b2-43596544fab6" (UID: "67845c4f-c0a5-464c-a9b2-43596544fab6"). InnerVolumeSpecName "kube-api-access-bzwzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:05:30.295072 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.294993 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67845c4f-c0a5-464c-a9b2-43596544fab6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67845c4f-c0a5-464c-a9b2-43596544fab6" (UID: "67845c4f-c0a5-464c-a9b2-43596544fab6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:05:30.393813 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.393726 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzwzs\" (UniqueName: \"kubernetes.io/projected/67845c4f-c0a5-464c-a9b2-43596544fab6-kube-api-access-bzwzs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:05:30.393813 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.393757 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67845c4f-c0a5-464c-a9b2-43596544fab6-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:05:30.393813 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.393769 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67845c4f-c0a5-464c-a9b2-43596544fab6-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:05:30.522818 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:30.522780 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:31.118988 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:31.118954 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" event={"ID":"67845c4f-c0a5-464c-a9b2-43596544fab6","Type":"ContainerDied","Data":"8799fefc4b2f08d3d808b6466bb147787b9e82884979edeb9c77dd11cac226f7"} Jul 17 18:05:31.118988 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:31.118982 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2" Jul 17 18:05:31.119498 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:31.119019 2573 scope.go:117] "RemoveContainer" containerID="a8bb27cfc15b1d9e8f6880530ce074e6a9f4fc14113850f1e8d137b564991838" Jul 17 18:05:31.144048 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:31.144015 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2"] Jul 17 18:05:31.151040 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:31.151016 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e49d4a9a-c4d5bb98f-9brf2"] Jul 17 18:05:31.778337 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:31.778296 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a36cfcb-66b1-4928-8497-f600d59f02f0" path="/var/lib/kubelet/pods/3a36cfcb-66b1-4928-8497-f600d59f02f0/volumes" Jul 17 18:05:31.778853 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:31.778830 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" path="/var/lib/kubelet/pods/67845c4f-c0a5-464c-a9b2-43596544fab6/volumes" Jul 17 18:05:34.500510 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:34.500476 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:36.573269 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:36.573240 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:36.674249 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:36.674213 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:37.104518 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:37.104471 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:37.218577 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:37.218541 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:38.757410 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:38.757372 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:39.552329 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.552284 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5"] Jul 17 18:05:39.552828 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.552797 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:05:39.552828 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.552830 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:05:39.553028 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.552842 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:05:39.553028 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.552849 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:05:39.553028 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.552874 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a36cfcb-66b1-4928-8497-f600d59f02f0" containerName="istio-proxy" Jul 17 18:05:39.553028 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.552883 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a36cfcb-66b1-4928-8497-f600d59f02f0" containerName="istio-proxy" Jul 17 18:05:39.553028 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.552906 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:05:39.553028 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.552914 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:05:39.553028 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.552928 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a36cfcb-66b1-4928-8497-f600d59f02f0" containerName="istio-proxy" Jul 17 18:05:39.553028 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.552936 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a36cfcb-66b1-4928-8497-f600d59f02f0" containerName="istio-proxy" Jul 17 18:05:39.553409 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.553036 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:05:39.553409 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.553051 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:05:39.553409 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.553063 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a36cfcb-66b1-4928-8497-f600d59f02f0" containerName="istio-proxy" Jul 17 18:05:39.553409 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.553073 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:05:39.553409 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.553080 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:05:39.553409 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.553089 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a36cfcb-66b1-4928-8497-f600d59f02f0" containerName="istio-proxy" Jul 17 18:05:39.553409 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.553097 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:05:39.557697 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.557676 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" Jul 17 18:05:39.561970 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.561949 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4f9n\"" Jul 17 18:05:39.576588 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.576555 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5"] Jul 17 18:05:39.581286 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.581254 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a0d0dc-319e-466e-8a51-664fc88d5124-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5\" (UID: \"a1a0d0dc-319e-466e-8a51-664fc88d5124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" Jul 17 18:05:39.682006 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.681972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a0d0dc-319e-466e-8a51-664fc88d5124-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5\" (UID: \"a1a0d0dc-319e-466e-8a51-664fc88d5124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" Jul 17 18:05:39.682565 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.682544 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a0d0dc-319e-466e-8a51-664fc88d5124-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5\" (UID: \"a1a0d0dc-319e-466e-8a51-664fc88d5124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" Jul 17 18:05:39.872541 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.872466 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4f9n\"" Jul 17 18:05:39.879834 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:39.879811 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" Jul 17 18:05:40.021137 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:40.021114 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5"] Jul 17 18:05:40.023329 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:05:40.023299 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a0d0dc_319e_466e_8a51_664fc88d5124.slice/crio-2a056ea115c4ff6095550af92ad3ec1caa2330719da64ece74346ee4c96d98d7 WatchSource:0}: Error finding container 2a056ea115c4ff6095550af92ad3ec1caa2330719da64ece74346ee4c96d98d7: Status 404 returned error can't find the container with id 2a056ea115c4ff6095550af92ad3ec1caa2330719da64ece74346ee4c96d98d7 Jul 17 18:05:40.172469 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:40.172380 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" event={"ID":"a1a0d0dc-319e-466e-8a51-664fc88d5124","Type":"ContainerStarted","Data":"2a056ea115c4ff6095550af92ad3ec1caa2330719da64ece74346ee4c96d98d7"} Jul 17 18:05:41.124679 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:41.124614 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:44.199058 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:44.198965 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" event={"ID":"a1a0d0dc-319e-466e-8a51-664fc88d5124","Type":"ContainerStarted","Data":"e5efeac97fc540bbe191976aa98c694d07a3806980273049f27c7c7091871fd0"} Jul 17 18:05:44.220554 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:44.220501 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" podStartSLOduration=1.594967721 podStartE2EDuration="5.220486623s" totalImagesPullingTime="3.625518902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:05:44.218189466 +0000 UTC m=+2464.953613515" watchObservedRunningTime="2026-07-17 18:05:44.220486623 +0000 UTC m=+2464.955910671" Jul 17 18:05:45.100772 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:45.100739 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:45.205469 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:45.205433 2573 generic.go:356] "Generic (PLEG): container finished" podID="a1a0d0dc-319e-466e-8a51-664fc88d5124" containerID="e5efeac97fc540bbe191976aa98c694d07a3806980273049f27c7c7091871fd0" exitCode=6 Jul 17 18:05:45.205469 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:45.205451 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:45.205926 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:45.205524 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" event={"ID":"a1a0d0dc-319e-466e-8a51-664fc88d5124","Type":"ContainerDied","Data":"e5efeac97fc540bbe191976aa98c694d07a3806980273049f27c7c7091871fd0"} Jul 17 18:05:45.205926 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:45.205730 2573 scope.go:117] "RemoveContainer" containerID="e5efeac97fc540bbe191976aa98c694d07a3806980273049f27c7c7091871fd0" Jul 17 18:05:45.322296 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:45.322260 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:45.436451 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:45.436418 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:45.923546 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:45.923517 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:46.214753 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:46.214668 2573 generic.go:356] "Generic (PLEG): container finished" podID="a1a0d0dc-319e-466e-8a51-664fc88d5124" containerID="3c4c32aa54df53b1c6a6260e75b9c327ca0d304a987dbb0f559dbdab1a9cdd10" exitCode=6 Jul 17 18:05:46.214753 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:46.214713 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" event={"ID":"a1a0d0dc-319e-466e-8a51-664fc88d5124","Type":"ContainerDied","Data":"3c4c32aa54df53b1c6a6260e75b9c327ca0d304a987dbb0f559dbdab1a9cdd10"} Jul 17 18:05:46.215244 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:46.214775 2573 scope.go:117] "RemoveContainer" containerID="e5efeac97fc540bbe191976aa98c694d07a3806980273049f27c7c7091871fd0" Jul 17 18:05:46.215244 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:46.214995 2573 scope.go:117] "RemoveContainer" containerID="3c4c32aa54df53b1c6a6260e75b9c327ca0d304a987dbb0f559dbdab1a9cdd10" Jul 17 18:05:46.215244 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:05:46.215214 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5_odh-ai-gateway-infra(a1a0d0dc-319e-466e-8a51-664fc88d5124)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" podUID="a1a0d0dc-319e-466e-8a51-664fc88d5124" Jul 17 18:05:47.221921 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:47.221891 2573 scope.go:117] "RemoveContainer" containerID="3c4c32aa54df53b1c6a6260e75b9c327ca0d304a987dbb0f559dbdab1a9cdd10" Jul 17 18:05:47.222299 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:05:47.222134 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5_odh-ai-gateway-infra(a1a0d0dc-319e-466e-8a51-664fc88d5124)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" podUID="a1a0d0dc-319e-466e-8a51-664fc88d5124" Jul 17 18:05:57.771879 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:57.771840 2573 scope.go:117] "RemoveContainer" containerID="3c4c32aa54df53b1c6a6260e75b9c327ca0d304a987dbb0f559dbdab1a9cdd10" Jul 17 18:05:58.277402 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:58.277316 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" event={"ID":"a1a0d0dc-319e-466e-8a51-664fc88d5124","Type":"ContainerStarted","Data":"b889ffddc0d400d3278a160183202446a1a4428b5139af1a0d05aa8c881bd141"} Jul 17 18:05:59.284397 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:59.284362 2573 generic.go:356] "Generic (PLEG): container finished" podID="a1a0d0dc-319e-466e-8a51-664fc88d5124" containerID="b889ffddc0d400d3278a160183202446a1a4428b5139af1a0d05aa8c881bd141" exitCode=6 Jul 17 18:05:59.284864 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:59.284437 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" event={"ID":"a1a0d0dc-319e-466e-8a51-664fc88d5124","Type":"ContainerDied","Data":"b889ffddc0d400d3278a160183202446a1a4428b5139af1a0d05aa8c881bd141"} Jul 17 18:05:59.284864 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:59.284500 2573 scope.go:117] "RemoveContainer" containerID="3c4c32aa54df53b1c6a6260e75b9c327ca0d304a987dbb0f559dbdab1a9cdd10" Jul 17 18:05:59.285014 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:59.284897 2573 scope.go:117] "RemoveContainer" containerID="b889ffddc0d400d3278a160183202446a1a4428b5139af1a0d05aa8c881bd141" Jul 17 18:05:59.285139 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:05:59.285117 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5_odh-ai-gateway-infra(a1a0d0dc-319e-466e-8a51-664fc88d5124)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" podUID="a1a0d0dc-319e-466e-8a51-664fc88d5124" Jul 17 18:05:59.310986 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:05:59.310950 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5"] Jul 17 18:06:00.428843 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:00.428817 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" Jul 17 18:06:00.584925 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:00.584820 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a0d0dc-319e-466e-8a51-664fc88d5124-maas-api-service-ca\") pod \"a1a0d0dc-319e-466e-8a51-664fc88d5124\" (UID: \"a1a0d0dc-319e-466e-8a51-664fc88d5124\") " Jul 17 18:06:00.585209 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:00.585184 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a0d0dc-319e-466e-8a51-664fc88d5124-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a1a0d0dc-319e-466e-8a51-664fc88d5124" (UID: "a1a0d0dc-319e-466e-8a51-664fc88d5124"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:06:00.686519 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:00.686471 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a0d0dc-319e-466e-8a51-664fc88d5124-maas-api-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:06:01.296589 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:01.296549 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" event={"ID":"a1a0d0dc-319e-466e-8a51-664fc88d5124","Type":"ContainerDied","Data":"2a056ea115c4ff6095550af92ad3ec1caa2330719da64ece74346ee4c96d98d7"} Jul 17 18:06:01.296589 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:01.296568 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5" Jul 17 18:06:01.296923 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:01.296609 2573 scope.go:117] "RemoveContainer" containerID="b889ffddc0d400d3278a160183202446a1a4428b5139af1a0d05aa8c881bd141" Jul 17 18:06:01.325976 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:01.325949 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5"] Jul 17 18:06:01.335783 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:01.335760 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-vxtb5"] Jul 17 18:06:01.777359 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:01.777327 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1a0d0dc-319e-466e-8a51-664fc88d5124" path="/var/lib/kubelet/pods/a1a0d0dc-319e-466e-8a51-664fc88d5124/volumes" Jul 17 18:06:04.098397 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.098349 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv"] Jul 17 18:06:04.098827 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.098760 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a0d0dc-319e-466e-8a51-664fc88d5124" containerName="revoke-keys" Jul 17 18:06:04.098827 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.098772 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a0d0dc-319e-466e-8a51-664fc88d5124" containerName="revoke-keys" Jul 17 18:06:04.098827 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.098790 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:06:04.098827 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.098795 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:06:04.098827 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.098810 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:06:04.098827 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.098816 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:06:04.098827 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.098824 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a0d0dc-319e-466e-8a51-664fc88d5124" containerName="revoke-keys" Jul 17 18:06:04.098827 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.098830 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a0d0dc-319e-466e-8a51-664fc88d5124" containerName="revoke-keys" Jul 17 18:06:04.099079 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.098888 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a0d0dc-319e-466e-8a51-664fc88d5124" containerName="revoke-keys" Jul 17 18:06:04.099079 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.098898 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:06:04.099079 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.098905 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a0d0dc-319e-466e-8a51-664fc88d5124" containerName="revoke-keys" Jul 17 18:06:04.099079 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.098911 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a0d0dc-319e-466e-8a51-664fc88d5124" containerName="revoke-keys" Jul 17 18:06:04.103490 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.103474 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" Jul 17 18:06:04.106667 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.106644 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4f9n\"" Jul 17 18:06:04.110825 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.110802 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv"] Jul 17 18:06:04.222743 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.222702 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/939e18a1-7048-481e-8778-65cfd7e99086-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv\" (UID: \"939e18a1-7048-481e-8778-65cfd7e99086\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" Jul 17 18:06:04.323972 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.323937 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/939e18a1-7048-481e-8778-65cfd7e99086-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv\" (UID: \"939e18a1-7048-481e-8778-65cfd7e99086\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" Jul 17 18:06:04.324587 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.324563 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/939e18a1-7048-481e-8778-65cfd7e99086-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv\" (UID: \"939e18a1-7048-481e-8778-65cfd7e99086\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" Jul 17 18:06:04.415865 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.415768 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" Jul 17 18:06:04.557395 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:04.557363 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv"] Jul 17 18:06:04.559172 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:06:04.559143 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939e18a1_7048_481e_8778_65cfd7e99086.slice/crio-bff329168037ca1e148d8a3430e93064a0a10ac446318b23aa6123c94fb369cb WatchSource:0}: Error finding container bff329168037ca1e148d8a3430e93064a0a10ac446318b23aa6123c94fb369cb: Status 404 returned error can't find the container with id bff329168037ca1e148d8a3430e93064a0a10ac446318b23aa6123c94fb369cb Jul 17 18:06:05.318837 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:05.318799 2573 generic.go:356] "Generic (PLEG): container finished" podID="939e18a1-7048-481e-8778-65cfd7e99086" containerID="cac63d528cff13d2a674410710561660c9f37e8fd73b6df5798b2b541f5fd7b2" exitCode=6 Jul 17 18:06:05.319314 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:05.318842 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" event={"ID":"939e18a1-7048-481e-8778-65cfd7e99086","Type":"ContainerDied","Data":"cac63d528cff13d2a674410710561660c9f37e8fd73b6df5798b2b541f5fd7b2"} Jul 17 18:06:05.319314 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:05.318912 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" event={"ID":"939e18a1-7048-481e-8778-65cfd7e99086","Type":"ContainerStarted","Data":"bff329168037ca1e148d8a3430e93064a0a10ac446318b23aa6123c94fb369cb"} Jul 17 18:06:05.319314 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:05.319170 2573 scope.go:117] "RemoveContainer" containerID="cac63d528cff13d2a674410710561660c9f37e8fd73b6df5798b2b541f5fd7b2" Jul 17 18:06:06.325330 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:06.325297 2573 generic.go:356] "Generic (PLEG): container finished" podID="939e18a1-7048-481e-8778-65cfd7e99086" containerID="b5b46c55d65c2dd836fb85ed83a847d80f48e6e733de6b1bd1d179f9b923f23b" exitCode=6 Jul 17 18:06:06.325818 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:06.325389 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" event={"ID":"939e18a1-7048-481e-8778-65cfd7e99086","Type":"ContainerDied","Data":"b5b46c55d65c2dd836fb85ed83a847d80f48e6e733de6b1bd1d179f9b923f23b"} Jul 17 18:06:06.325818 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:06.325453 2573 scope.go:117] "RemoveContainer" containerID="cac63d528cff13d2a674410710561660c9f37e8fd73b6df5798b2b541f5fd7b2" Jul 17 18:06:06.325818 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:06.325596 2573 scope.go:117] "RemoveContainer" containerID="b5b46c55d65c2dd836fb85ed83a847d80f48e6e733de6b1bd1d179f9b923f23b" Jul 17 18:06:06.325936 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:06:06.325915 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv_odh-ai-gateway-infra(939e18a1-7048-481e-8778-65cfd7e99086)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" podUID="939e18a1-7048-481e-8778-65cfd7e99086" Jul 17 18:06:07.332209 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:07.332167 2573 scope.go:117] "RemoveContainer" containerID="b5b46c55d65c2dd836fb85ed83a847d80f48e6e733de6b1bd1d179f9b923f23b" Jul 17 18:06:07.332610 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:06:07.332397 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv_odh-ai-gateway-infra(939e18a1-7048-481e-8778-65cfd7e99086)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" podUID="939e18a1-7048-481e-8778-65cfd7e99086" Jul 17 18:06:11.837936 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:11.837900 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:16.432636 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:16.432592 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7"] Jul 17 18:06:16.433041 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:16.433028 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:06:16.433085 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:16.433044 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="67845c4f-c0a5-464c-a9b2-43596544fab6" containerName="maas-api" Jul 17 18:06:16.433085 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:16.433063 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a0d0dc-319e-466e-8a51-664fc88d5124" containerName="revoke-keys" Jul 17 18:06:16.433085 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:16.433068 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a0d0dc-319e-466e-8a51-664fc88d5124" containerName="revoke-keys" Jul 17 18:06:16.435777 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:16.435762 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" Jul 17 18:06:16.437180 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:16.437147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78385db4-db65-475c-b524-98e75adf6c84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-59516a0f-db6d7\" (UID: \"78385db4-db65-475c-b524-98e75adf6c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" Jul 17 18:06:16.444972 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:16.444946 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7"] Jul 17 18:06:16.538157 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:16.538119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78385db4-db65-475c-b524-98e75adf6c84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-59516a0f-db6d7\" (UID: \"78385db4-db65-475c-b524-98e75adf6c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" Jul 17 18:06:16.538808 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:16.538790 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78385db4-db65-475c-b524-98e75adf6c84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-59516a0f-db6d7\" (UID: \"78385db4-db65-475c-b524-98e75adf6c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" Jul 17 18:06:16.746265 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:16.746227 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" Jul 17 18:06:16.880933 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:16.880902 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7"] Jul 17 18:06:16.882867 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:06:16.882840 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78385db4_db65_475c_b524_98e75adf6c84.slice/crio-9dbac7abf005801b7c8a9e488b0ac2f716bf9bfc19d93c678fc359d5dc9f2cb7 WatchSource:0}: Error finding container 9dbac7abf005801b7c8a9e488b0ac2f716bf9bfc19d93c678fc359d5dc9f2cb7: Status 404 returned error can't find the container with id 9dbac7abf005801b7c8a9e488b0ac2f716bf9bfc19d93c678fc359d5dc9f2cb7 Jul 17 18:06:17.383928 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:17.383829 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" event={"ID":"78385db4-db65-475c-b524-98e75adf6c84","Type":"ContainerStarted","Data":"4defad6a3d7a03932872a09a85438c534eecba41c22dbf47c84f52757ccba77e"} Jul 17 18:06:17.383928 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:17.383882 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" event={"ID":"78385db4-db65-475c-b524-98e75adf6c84","Type":"ContainerStarted","Data":"9dbac7abf005801b7c8a9e488b0ac2f716bf9bfc19d93c678fc359d5dc9f2cb7"} Jul 17 18:06:17.403002 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:17.402942 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" podStartSLOduration=1.40292093 podStartE2EDuration="1.40292093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:06:17.401373465 +0000 UTC m=+2498.136797516" watchObservedRunningTime="2026-07-17 18:06:17.40292093 +0000 UTC m=+2498.138344981" Jul 17 18:06:17.960822 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:17.960785 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:18.065899 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:18.065865 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:18.162561 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:18.162532 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:18.390332 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:18.390296 2573 generic.go:356] "Generic (PLEG): container finished" podID="78385db4-db65-475c-b524-98e75adf6c84" containerID="4defad6a3d7a03932872a09a85438c534eecba41c22dbf47c84f52757ccba77e" exitCode=6 Jul 17 18:06:18.390527 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:18.390338 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" event={"ID":"78385db4-db65-475c-b524-98e75adf6c84","Type":"ContainerDied","Data":"4defad6a3d7a03932872a09a85438c534eecba41c22dbf47c84f52757ccba77e"} Jul 17 18:06:18.390615 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:18.390600 2573 scope.go:117] "RemoveContainer" containerID="4defad6a3d7a03932872a09a85438c534eecba41c22dbf47c84f52757ccba77e" Jul 17 18:06:19.397695 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:19.397577 2573 generic.go:356] "Generic (PLEG): container finished" podID="78385db4-db65-475c-b524-98e75adf6c84" containerID="64d51942a2b5c33576133cfec03c1ce14256b836edbda9a9edd76b6294995b48" exitCode=6 Jul 17 18:06:19.397695 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:19.397668 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" event={"ID":"78385db4-db65-475c-b524-98e75adf6c84","Type":"ContainerDied","Data":"64d51942a2b5c33576133cfec03c1ce14256b836edbda9a9edd76b6294995b48"} Jul 17 18:06:19.398157 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:19.397740 2573 scope.go:117] "RemoveContainer" containerID="4defad6a3d7a03932872a09a85438c534eecba41c22dbf47c84f52757ccba77e" Jul 17 18:06:19.398157 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:19.397911 2573 scope.go:117] "RemoveContainer" containerID="64d51942a2b5c33576133cfec03c1ce14256b836edbda9a9edd76b6294995b48" Jul 17 18:06:19.398229 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:06:19.398166 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-59516a0f-db6d7_odh-ai-gateway-infra(78385db4-db65-475c-b524-98e75adf6c84)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" podUID="78385db4-db65-475c-b524-98e75adf6c84" Jul 17 18:06:19.781091 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:19.781060 2573 scope.go:117] "RemoveContainer" containerID="b5b46c55d65c2dd836fb85ed83a847d80f48e6e733de6b1bd1d179f9b923f23b" Jul 17 18:06:20.404316 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:20.404286 2573 scope.go:117] "RemoveContainer" containerID="64d51942a2b5c33576133cfec03c1ce14256b836edbda9a9edd76b6294995b48" Jul 17 18:06:20.404835 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:06:20.404572 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-59516a0f-db6d7_odh-ai-gateway-infra(78385db4-db65-475c-b524-98e75adf6c84)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" podUID="78385db4-db65-475c-b524-98e75adf6c84" Jul 17 18:06:20.405944 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:20.405915 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" event={"ID":"939e18a1-7048-481e-8778-65cfd7e99086","Type":"ContainerStarted","Data":"cca2cc7c0e7143da3d5ed8e0cb5cc8c5bcb249209d5e3d3ee73d5334af143c0e"} Jul 17 18:06:20.440893 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:20.440837 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" podStartSLOduration=16.440823102 podStartE2EDuration="16.440823102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:06:20.439726178 +0000 UTC m=+2501.175150227" watchObservedRunningTime="2026-07-17 18:06:20.440823102 +0000 UTC m=+2501.176247183" Jul 17 18:06:21.412313 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:21.412274 2573 generic.go:356] "Generic (PLEG): container finished" podID="939e18a1-7048-481e-8778-65cfd7e99086" containerID="cca2cc7c0e7143da3d5ed8e0cb5cc8c5bcb249209d5e3d3ee73d5334af143c0e" exitCode=6 Jul 17 18:06:21.412838 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:21.412349 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" event={"ID":"939e18a1-7048-481e-8778-65cfd7e99086","Type":"ContainerDied","Data":"cca2cc7c0e7143da3d5ed8e0cb5cc8c5bcb249209d5e3d3ee73d5334af143c0e"} Jul 17 18:06:21.412838 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:21.412410 2573 scope.go:117] "RemoveContainer" containerID="b5b46c55d65c2dd836fb85ed83a847d80f48e6e733de6b1bd1d179f9b923f23b" Jul 17 18:06:21.412838 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:21.412752 2573 scope.go:117] "RemoveContainer" containerID="cca2cc7c0e7143da3d5ed8e0cb5cc8c5bcb249209d5e3d3ee73d5334af143c0e" Jul 17 18:06:21.413045 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:06:21.413022 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv_odh-ai-gateway-infra(939e18a1-7048-481e-8778-65cfd7e99086)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" podUID="939e18a1-7048-481e-8778-65cfd7e99086" Jul 17 18:06:21.451805 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:21.451766 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv"] Jul 17 18:06:22.551909 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:22.551881 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" Jul 17 18:06:22.595364 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:22.595328 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/939e18a1-7048-481e-8778-65cfd7e99086-maas-api-service-ca\") pod \"939e18a1-7048-481e-8778-65cfd7e99086\" (UID: \"939e18a1-7048-481e-8778-65cfd7e99086\") " Jul 17 18:06:22.595667 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:22.595646 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939e18a1-7048-481e-8778-65cfd7e99086-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "939e18a1-7048-481e-8778-65cfd7e99086" (UID: "939e18a1-7048-481e-8778-65cfd7e99086"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:06:22.696284 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:22.696195 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/939e18a1-7048-481e-8778-65cfd7e99086-maas-api-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:06:23.425303 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:23.425270 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" event={"ID":"939e18a1-7048-481e-8778-65cfd7e99086","Type":"ContainerDied","Data":"bff329168037ca1e148d8a3430e93064a0a10ac446318b23aa6123c94fb369cb"} Jul 17 18:06:23.425490 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:23.425333 2573 scope.go:117] "RemoveContainer" containerID="cca2cc7c0e7143da3d5ed8e0cb5cc8c5bcb249209d5e3d3ee73d5334af143c0e" Jul 17 18:06:23.425490 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:23.425279 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv" Jul 17 18:06:23.454012 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:23.453981 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv"] Jul 17 18:06:23.458119 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:23.458093 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-j5rqv"] Jul 17 18:06:23.778419 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:23.778389 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="939e18a1-7048-481e-8778-65cfd7e99086" path="/var/lib/kubelet/pods/939e18a1-7048-481e-8778-65cfd7e99086/volumes" Jul 17 18:06:30.828585 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:30.828510 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:32.771800 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:32.771763 2573 scope.go:117] "RemoveContainer" containerID="64d51942a2b5c33576133cfec03c1ce14256b836edbda9a9edd76b6294995b48" Jul 17 18:06:33.476300 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:33.476266 2573 generic.go:356] "Generic (PLEG): container finished" podID="78385db4-db65-475c-b524-98e75adf6c84" containerID="55eecef60b525195fd19884205544245eaefeb2f7c72586d95fb433d12d7d3da" exitCode=6 Jul 17 18:06:33.476439 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:33.476336 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" event={"ID":"78385db4-db65-475c-b524-98e75adf6c84","Type":"ContainerDied","Data":"55eecef60b525195fd19884205544245eaefeb2f7c72586d95fb433d12d7d3da"} Jul 17 18:06:33.476439 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:33.476397 2573 scope.go:117] "RemoveContainer" containerID="64d51942a2b5c33576133cfec03c1ce14256b836edbda9a9edd76b6294995b48" Jul 17 18:06:33.476758 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:33.476740 2573 scope.go:117] "RemoveContainer" containerID="55eecef60b525195fd19884205544245eaefeb2f7c72586d95fb433d12d7d3da" Jul 17 18:06:33.477033 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:06:33.477010 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-59516a0f-db6d7_odh-ai-gateway-infra(78385db4-db65-475c-b524-98e75adf6c84)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" podUID="78385db4-db65-475c-b524-98e75adf6c84" Jul 17 18:06:34.036381 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:34.036350 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:34.503568 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:34.503527 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7"] Jul 17 18:06:34.637740 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:34.637708 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" Jul 17 18:06:34.720756 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:34.720714 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78385db4-db65-475c-b524-98e75adf6c84-maas-api-service-ca\") pod \"78385db4-db65-475c-b524-98e75adf6c84\" (UID: \"78385db4-db65-475c-b524-98e75adf6c84\") " Jul 17 18:06:34.721110 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:34.721085 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78385db4-db65-475c-b524-98e75adf6c84-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78385db4-db65-475c-b524-98e75adf6c84" (UID: "78385db4-db65-475c-b524-98e75adf6c84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:06:34.822568 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:34.822485 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78385db4-db65-475c-b524-98e75adf6c84-maas-api-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:06:35.488928 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:35.488897 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" Jul 17 18:06:35.488928 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:35.488909 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7" event={"ID":"78385db4-db65-475c-b524-98e75adf6c84","Type":"ContainerDied","Data":"9dbac7abf005801b7c8a9e488b0ac2f716bf9bfc19d93c678fc359d5dc9f2cb7"} Jul 17 18:06:35.489436 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:35.488965 2573 scope.go:117] "RemoveContainer" containerID="55eecef60b525195fd19884205544245eaefeb2f7c72586d95fb433d12d7d3da" Jul 17 18:06:35.511231 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:35.511197 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7"] Jul 17 18:06:35.517909 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:35.517886 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-db6d7"] Jul 17 18:06:35.777247 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:35.777163 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78385db4-db65-475c-b524-98e75adf6c84" path="/var/lib/kubelet/pods/78385db4-db65-475c-b524-98e75adf6c84/volumes" Jul 17 18:06:40.236825 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:40.236786 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:40.345148 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:40.345071 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:40.496861 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:40.496822 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:50.995239 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:50.995196 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:51.783679 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:51.783649 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:57.869418 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:57.869384 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:57.977322 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:57.977283 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:58.083903 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:58.083875 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:58.180588 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:58.180498 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:58.955811 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:58.955779 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:59.061659 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:06:59.061595 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:00.706197 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:00.706160 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:06.473037 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.472999 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd"] Jul 17 18:07:06.473471 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.473405 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78385db4-db65-475c-b524-98e75adf6c84" containerName="revoke-keys" Jul 17 18:07:06.473471 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.473417 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="78385db4-db65-475c-b524-98e75adf6c84" containerName="revoke-keys" Jul 17 18:07:06.473471 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.473434 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78385db4-db65-475c-b524-98e75adf6c84" containerName="revoke-keys" Jul 17 18:07:06.473471 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.473440 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="78385db4-db65-475c-b524-98e75adf6c84" containerName="revoke-keys" Jul 17 18:07:06.473471 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.473455 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="939e18a1-7048-481e-8778-65cfd7e99086" containerName="revoke-keys" Jul 17 18:07:06.473471 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.473461 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="939e18a1-7048-481e-8778-65cfd7e99086" containerName="revoke-keys" Jul 17 18:07:06.473471 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.473471 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="939e18a1-7048-481e-8778-65cfd7e99086" containerName="revoke-keys" Jul 17 18:07:06.473745 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.473478 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="939e18a1-7048-481e-8778-65cfd7e99086" containerName="revoke-keys" Jul 17 18:07:06.473745 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.473536 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="939e18a1-7048-481e-8778-65cfd7e99086" containerName="revoke-keys" Jul 17 18:07:06.473745 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.473542 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="939e18a1-7048-481e-8778-65cfd7e99086" containerName="revoke-keys" Jul 17 18:07:06.473745 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.473549 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="78385db4-db65-475c-b524-98e75adf6c84" containerName="revoke-keys" Jul 17 18:07:06.473745 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.473555 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="78385db4-db65-475c-b524-98e75adf6c84" containerName="revoke-keys" Jul 17 18:07:06.476506 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.476490 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" Jul 17 18:07:06.479379 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.479358 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4f9n\"" Jul 17 18:07:06.485603 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.485577 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd"] Jul 17 18:07:06.534179 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.534136 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7c2cba-034d-412b-8396-61aed8d1d9bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd\" (UID: \"6d7c2cba-034d-412b-8396-61aed8d1d9bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" Jul 17 18:07:06.576253 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.576220 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:06.635426 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.635392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7c2cba-034d-412b-8396-61aed8d1d9bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd\" (UID: \"6d7c2cba-034d-412b-8396-61aed8d1d9bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" Jul 17 18:07:06.636046 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.636027 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7c2cba-034d-412b-8396-61aed8d1d9bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd\" (UID: \"6d7c2cba-034d-412b-8396-61aed8d1d9bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" Jul 17 18:07:06.643151 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.643127 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:06.788165 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.788129 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" Jul 17 18:07:06.919862 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:06.919828 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd"] Jul 17 18:07:06.921473 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:07:06.921448 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7c2cba_034d_412b_8396_61aed8d1d9bb.slice/crio-b0d0c2640e83c4c102900b4ad73070dd03cf135eec47c294ed48e9176358d072 WatchSource:0}: Error finding container b0d0c2640e83c4c102900b4ad73070dd03cf135eec47c294ed48e9176358d072: Status 404 returned error can't find the container with id b0d0c2640e83c4c102900b4ad73070dd03cf135eec47c294ed48e9176358d072 Jul 17 18:07:07.257615 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:07.257580 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:07.645855 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:07.645821 2573 generic.go:356] "Generic (PLEG): container finished" podID="6d7c2cba-034d-412b-8396-61aed8d1d9bb" containerID="882848c19be0d852f4c5df34c4073e4648c2900545f9bfa41ca80cd4f9b00bfc" exitCode=6 Jul 17 18:07:07.646264 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:07.645873 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" event={"ID":"6d7c2cba-034d-412b-8396-61aed8d1d9bb","Type":"ContainerDied","Data":"882848c19be0d852f4c5df34c4073e4648c2900545f9bfa41ca80cd4f9b00bfc"} Jul 17 18:07:07.646264 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:07.645917 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" event={"ID":"6d7c2cba-034d-412b-8396-61aed8d1d9bb","Type":"ContainerStarted","Data":"b0d0c2640e83c4c102900b4ad73070dd03cf135eec47c294ed48e9176358d072"} Jul 17 18:07:07.646264 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:07.646110 2573 scope.go:117] "RemoveContainer" containerID="882848c19be0d852f4c5df34c4073e4648c2900545f9bfa41ca80cd4f9b00bfc" Jul 17 18:07:08.046731 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.046694 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw"] Jul 17 18:07:08.047163 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.047150 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="939e18a1-7048-481e-8778-65cfd7e99086" containerName="revoke-keys" Jul 17 18:07:08.047208 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.047165 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="939e18a1-7048-481e-8778-65cfd7e99086" containerName="revoke-keys" Jul 17 18:07:08.047208 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.047185 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78385db4-db65-475c-b524-98e75adf6c84" containerName="revoke-keys" Jul 17 18:07:08.047208 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.047191 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="78385db4-db65-475c-b524-98e75adf6c84" containerName="revoke-keys" Jul 17 18:07:08.047298 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.047259 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="78385db4-db65-475c-b524-98e75adf6c84" containerName="revoke-keys" Jul 17 18:07:08.047298 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.047277 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="939e18a1-7048-481e-8778-65cfd7e99086" containerName="revoke-keys" Jul 17 18:07:08.050439 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.050413 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw" Jul 17 18:07:08.077785 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.077747 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw"] Jul 17 18:07:08.151126 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.151082 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6a7d4f3-87c2-46f4-82d0-eee80578ac6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw\" (UID: \"c6a7d4f3-87c2-46f4-82d0-eee80578ac6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw" Jul 17 18:07:08.252327 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.252283 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6a7d4f3-87c2-46f4-82d0-eee80578ac6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw\" (UID: \"c6a7d4f3-87c2-46f4-82d0-eee80578ac6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw" Jul 17 18:07:08.253015 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.252994 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6a7d4f3-87c2-46f4-82d0-eee80578ac6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw\" (UID: \"c6a7d4f3-87c2-46f4-82d0-eee80578ac6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw" Jul 17 18:07:08.361121 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.361091 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw" Jul 17 18:07:08.495961 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.495932 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw"] Jul 17 18:07:08.497588 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:07:08.497561 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a7d4f3_87c2_46f4_82d0_eee80578ac6d.slice/crio-49de90f16050aafa2bcad5e728e60c82b276c17292b378dddae2f861e12398d6 WatchSource:0}: Error finding container 49de90f16050aafa2bcad5e728e60c82b276c17292b378dddae2f861e12398d6: Status 404 returned error can't find the container with id 49de90f16050aafa2bcad5e728e60c82b276c17292b378dddae2f861e12398d6 Jul 17 18:07:08.652396 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.652360 2573 generic.go:356] "Generic (PLEG): container finished" podID="6d7c2cba-034d-412b-8396-61aed8d1d9bb" containerID="1227937bbd40c6eccd56671471cbdaa55ca55576856c34b5c3578fe9df6fda74" exitCode=6 Jul 17 18:07:08.652839 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.652445 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" event={"ID":"6d7c2cba-034d-412b-8396-61aed8d1d9bb","Type":"ContainerDied","Data":"1227937bbd40c6eccd56671471cbdaa55ca55576856c34b5c3578fe9df6fda74"} Jul 17 18:07:08.652839 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.652510 2573 scope.go:117] "RemoveContainer" containerID="882848c19be0d852f4c5df34c4073e4648c2900545f9bfa41ca80cd4f9b00bfc" Jul 17 18:07:08.652839 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.652737 2573 scope.go:117] "RemoveContainer" containerID="1227937bbd40c6eccd56671471cbdaa55ca55576856c34b5c3578fe9df6fda74" Jul 17 18:07:08.653007 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:07:08.652986 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd_odh-ai-gateway-infra(6d7c2cba-034d-412b-8396-61aed8d1d9bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" podUID="6d7c2cba-034d-412b-8396-61aed8d1d9bb" Jul 17 18:07:08.653985 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:08.653963 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw" event={"ID":"c6a7d4f3-87c2-46f4-82d0-eee80578ac6d","Type":"ContainerStarted","Data":"49de90f16050aafa2bcad5e728e60c82b276c17292b378dddae2f861e12398d6"} Jul 17 18:07:09.660478 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:09.660442 2573 generic.go:356] "Generic (PLEG): container finished" podID="c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" containerID="af34764c89022f6119d77a94d241200607ad3be3ca8bea0afae48f4c2a14706d" exitCode=6 Jul 17 18:07:09.660959 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:09.660533 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw" event={"ID":"c6a7d4f3-87c2-46f4-82d0-eee80578ac6d","Type":"ContainerDied","Data":"af34764c89022f6119d77a94d241200607ad3be3ca8bea0afae48f4c2a14706d"} Jul 17 18:07:09.660959 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:09.660745 2573 scope.go:117] "RemoveContainer" containerID="af34764c89022f6119d77a94d241200607ad3be3ca8bea0afae48f4c2a14706d" Jul 17 18:07:09.662379 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:09.662361 2573 scope.go:117] "RemoveContainer" containerID="1227937bbd40c6eccd56671471cbdaa55ca55576856c34b5c3578fe9df6fda74" Jul 17 18:07:09.662572 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:07:09.662552 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd_odh-ai-gateway-infra(6d7c2cba-034d-412b-8396-61aed8d1d9bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" podUID="6d7c2cba-034d-412b-8396-61aed8d1d9bb" Jul 17 18:07:10.669243 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:10.669205 2573 generic.go:356] "Generic (PLEG): container finished" podID="c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" containerID="1534b7bca3114e0066ce63ff1a1dc9335e911a6fc6d840f29f19a6bfa32820d2" exitCode=6 Jul 17 18:07:10.669720 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:10.669278 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw" event={"ID":"c6a7d4f3-87c2-46f4-82d0-eee80578ac6d","Type":"ContainerDied","Data":"1534b7bca3114e0066ce63ff1a1dc9335e911a6fc6d840f29f19a6bfa32820d2"} Jul 17 18:07:10.669720 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:10.669338 2573 scope.go:117] "RemoveContainer" containerID="af34764c89022f6119d77a94d241200607ad3be3ca8bea0afae48f4c2a14706d" Jul 17 18:07:10.669720 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:10.669556 2573 scope.go:117] "RemoveContainer" containerID="1534b7bca3114e0066ce63ff1a1dc9335e911a6fc6d840f29f19a6bfa32820d2" Jul 17 18:07:10.669858 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:07:10.669817 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw_odh-ai-gateway-infra(c6a7d4f3-87c2-46f4-82d0-eee80578ac6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw" podUID="c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" Jul 17 18:07:11.676734 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:11.676702 2573 scope.go:117] "RemoveContainer" containerID="1534b7bca3114e0066ce63ff1a1dc9335e911a6fc6d840f29f19a6bfa32820d2" Jul 17 18:07:11.677131 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:07:11.676907 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw_odh-ai-gateway-infra(c6a7d4f3-87c2-46f4-82d0-eee80578ac6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw" podUID="c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" Jul 17 18:07:13.126015 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.125977 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:13.232186 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.232148 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:13.402992 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.402901 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:13.695396 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.695311 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246"] Jul 17 18:07:13.700454 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.700427 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.703018 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.702995 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f47fb1c3-openshift-default-dockercfg-jgtcv\"" Jul 17 18:07:13.715116 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.715091 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246"] Jul 17 18:07:13.736049 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.736024 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:13.816227 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.816185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f73b8a6-4553-4687-9149-e466ba680e15-istiod-ca-cert\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.816431 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.816254 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-istio-envoy\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.816431 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.816342 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpvfl\" (UniqueName: \"kubernetes.io/projected/6f73b8a6-4553-4687-9149-e466ba680e15-kube-api-access-hpvfl\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.816561 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.816429 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f73b8a6-4553-4687-9149-e466ba680e15-istio-token\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.816561 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.816492 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-credential-socket\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.816705 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.816568 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-workload-socket\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.816705 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.816607 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f73b8a6-4553-4687-9149-e466ba680e15-istio-podinfo\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.816705 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.816658 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-istio-data\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.816820 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.816708 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-workload-certs\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.917811 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.917775 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-istio-envoy\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.917977 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.917830 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpvfl\" (UniqueName: \"kubernetes.io/projected/6f73b8a6-4553-4687-9149-e466ba680e15-kube-api-access-hpvfl\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.917977 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.917860 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f73b8a6-4553-4687-9149-e466ba680e15-istio-token\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.917977 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.917890 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-credential-socket\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.917977 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.917923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-workload-socket\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.917977 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.917946 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f73b8a6-4553-4687-9149-e466ba680e15-istio-podinfo\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.917977 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.917962 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-istio-data\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.918339 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.917992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-workload-certs\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.918339 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.918014 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f73b8a6-4553-4687-9149-e466ba680e15-istiod-ca-cert\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.918339 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.918302 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-credential-socket\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.918525 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.918354 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-istio-data\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.918525 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.918382 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-workload-socket\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.918646 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.918570 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-workload-certs\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.918717 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.918699 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f73b8a6-4553-4687-9149-e466ba680e15-istiod-ca-cert\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.920135 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.920107 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-istio-envoy\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.920374 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.920353 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f73b8a6-4553-4687-9149-e466ba680e15-istio-podinfo\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.929387 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.929368 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f73b8a6-4553-4687-9149-e466ba680e15-istio-token\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:13.938428 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:13.938398 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpvfl\" (UniqueName: \"kubernetes.io/projected/6f73b8a6-4553-4687-9149-e466ba680e15-kube-api-access-hpvfl\") pod \"e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:14.011189 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:14.011150 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:14.175913 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:14.175881 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:14.198057 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:14.198027 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246"] Jul 17 18:07:14.201954 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:07:14.201926 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f73b8a6_4553_4687_9149_e466ba680e15.slice/crio-478416c3d4f528f551d11fbe31418825930a608514a136ab602867b8a45a0adb WatchSource:0}: Error finding container 478416c3d4f528f551d11fbe31418825930a608514a136ab602867b8a45a0adb: Status 404 returned error can't find the container with id 478416c3d4f528f551d11fbe31418825930a608514a136ab602867b8a45a0adb Jul 17 18:07:14.204048 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:14.204015 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:07:14.204107 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:14.204083 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:07:14.204148 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:14.204110 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:07:14.693258 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:14.693222 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" event={"ID":"6f73b8a6-4553-4687-9149-e466ba680e15","Type":"ContainerStarted","Data":"2874f07204e95204af0d963f56e4de05053cafd1d55b2e66f6e26d68b6b5a6f2"} Jul 17 18:07:14.693467 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:14.693288 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" event={"ID":"6f73b8a6-4553-4687-9149-e466ba680e15","Type":"ContainerStarted","Data":"478416c3d4f528f551d11fbe31418825930a608514a136ab602867b8a45a0adb"} Jul 17 18:07:14.717078 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:14.717023 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" podStartSLOduration=1.717006808 podStartE2EDuration="1.717006808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:07:14.71544753 +0000 UTC m=+2555.450871607" watchObservedRunningTime="2026-07-17 18:07:14.717006808 +0000 UTC m=+2555.452430930" Jul 17 18:07:15.012139 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:15.012094 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:15.017718 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:15.017685 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:15.697990 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:15.697959 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:15.699138 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:15.699120 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:16.763051 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:16.762988 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:16.839788 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:16.839751 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj"] Jul 17 18:07:16.843011 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:16.842990 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" Jul 17 18:07:16.847759 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:16.847734 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f47fb1c3-serving-cert\"" Jul 17 18:07:16.861318 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:16.861294 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj"] Jul 17 18:07:16.868822 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:16.868796 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:16.949717 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:16.949677 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-openshift-service-ca\") pod \"maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj\" (UID: \"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" Jul 17 18:07:16.949911 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:16.949752 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxhsb\" (UniqueName: \"kubernetes.io/projected/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-kube-api-access-qxhsb\") pod \"maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj\" (UID: \"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" Jul 17 18:07:16.949994 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:16.949960 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-maas-api-tls\") pod \"maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj\" (UID: \"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" Jul 17 18:07:16.965172 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:16.965139 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:17.051533 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:17.051443 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-openshift-service-ca\") pod \"maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj\" (UID: \"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" Jul 17 18:07:17.051533 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:17.051487 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhsb\" (UniqueName: \"kubernetes.io/projected/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-kube-api-access-qxhsb\") pod \"maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj\" (UID: \"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" Jul 17 18:07:17.051745 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:17.051551 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-maas-api-tls\") pod \"maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj\" (UID: \"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" Jul 17 18:07:17.051745 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:07:17.051700 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-serving-cert: secret "maas-api-e2e-mt-f47fb1c3-serving-cert" not found Jul 17 18:07:17.051817 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:07:17.051756 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-maas-api-tls podName:bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6 nodeName:}" failed. No retries permitted until 2026-07-17 18:07:17.551739972 +0000 UTC m=+2558.287163998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-maas-api-tls") pod "maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" (UID: "bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6") : secret "maas-api-e2e-mt-f47fb1c3-serving-cert" not found Jul 17 18:07:17.052197 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:17.052178 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-openshift-service-ca\") pod \"maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj\" (UID: \"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" Jul 17 18:07:17.066900 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:17.066875 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxhsb\" (UniqueName: \"kubernetes.io/projected/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-kube-api-access-qxhsb\") pod \"maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj\" (UID: \"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" Jul 17 18:07:17.402741 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:17.402646 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:17.556729 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:17.556694 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-maas-api-tls\") pod \"maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj\" (UID: \"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" Jul 17 18:07:17.559190 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:17.559169 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-maas-api-tls\") pod \"maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj\" (UID: \"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" Jul 17 18:07:17.561302 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:17.561281 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:17.663805 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:17.663719 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:17.754486 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:17.754451 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" Jul 17 18:07:17.905852 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:17.905825 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj"] Jul 17 18:07:17.907505 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:07:17.907468 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc45eed_8dbb_4d90_9e03_aaaf2ce2abb6.slice/crio-122ffb396665f8fdca79f9ab1bf5d0780c1495f54a59e98f04fd9cd4e5dd9c1b WatchSource:0}: Error finding container 122ffb396665f8fdca79f9ab1bf5d0780c1495f54a59e98f04fd9cd4e5dd9c1b: Status 404 returned error can't find the container with id 122ffb396665f8fdca79f9ab1bf5d0780c1495f54a59e98f04fd9cd4e5dd9c1b Jul 17 18:07:17.909206 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:17.909186 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:07:18.715465 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:18.715373 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" event={"ID":"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6","Type":"ContainerStarted","Data":"19bc7812a8c6de0c8de03b74e9d171dbce704ece21f7ac86d5ab47e558c0c16c"} Jul 17 18:07:18.715465 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:18.715426 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" event={"ID":"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6","Type":"ContainerStarted","Data":"122ffb396665f8fdca79f9ab1bf5d0780c1495f54a59e98f04fd9cd4e5dd9c1b"} Jul 17 18:07:18.715717 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:18.715470 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" Jul 17 18:07:18.736173 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:18.736123 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" podStartSLOduration=2.29173238 podStartE2EDuration="2.736106817s" totalImagesPullingTime="444.374437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:07:18.735342632 +0000 UTC m=+2559.470766683" watchObservedRunningTime="2026-07-17 18:07:18.736106817 +0000 UTC m=+2559.471530868" Jul 17 18:07:21.321972 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:21.321934 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:22.157658 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:22.157600 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:22.251928 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:22.251893 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:22.525429 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:22.525394 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:22.771557 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:22.771523 2573 scope.go:117] "RemoveContainer" containerID="1534b7bca3114e0066ce63ff1a1dc9335e911a6fc6d840f29f19a6bfa32820d2" Jul 17 18:07:22.771755 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:22.771605 2573 scope.go:117] "RemoveContainer" containerID="1227937bbd40c6eccd56671471cbdaa55ca55576856c34b5c3578fe9df6fda74" Jul 17 18:07:23.743262 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:23.743221 2573 generic.go:356] "Generic (PLEG): container finished" podID="c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" containerID="01604193cccecfa89ecf5fe713caa4bbbe6b56ec17f2de92a16641c9a147e5d8" exitCode=6 Jul 17 18:07:23.743727 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:23.743294 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw" event={"ID":"c6a7d4f3-87c2-46f4-82d0-eee80578ac6d","Type":"ContainerDied","Data":"01604193cccecfa89ecf5fe713caa4bbbe6b56ec17f2de92a16641c9a147e5d8"} Jul 17 18:07:23.743727 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:23.743400 2573 scope.go:117] "RemoveContainer" containerID="1534b7bca3114e0066ce63ff1a1dc9335e911a6fc6d840f29f19a6bfa32820d2" Jul 17 18:07:23.743853 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:23.743732 2573 scope.go:117] "RemoveContainer" containerID="01604193cccecfa89ecf5fe713caa4bbbe6b56ec17f2de92a16641c9a147e5d8" Jul 17 18:07:23.743984 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:07:23.743963 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw_odh-ai-gateway-infra(c6a7d4f3-87c2-46f4-82d0-eee80578ac6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw" podUID="c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" Jul 17 18:07:23.745376 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:23.745356 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" event={"ID":"6d7c2cba-034d-412b-8396-61aed8d1d9bb","Type":"ContainerStarted","Data":"76482105fa65eec44e01a3aeb98a6b5d33238068fd8b5c4356111b9ef3de2e1a"} Jul 17 18:07:23.777637 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:23.777566 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" podStartSLOduration=17.777550466 podStartE2EDuration="17.777550466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:07:23.775805531 +0000 UTC m=+2564.511229581" watchObservedRunningTime="2026-07-17 18:07:23.777550466 +0000 UTC m=+2564.512974515" Jul 17 18:07:24.729075 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:24.729043 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" Jul 17 18:07:24.753987 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:24.753957 2573 generic.go:356] "Generic (PLEG): container finished" podID="6d7c2cba-034d-412b-8396-61aed8d1d9bb" containerID="76482105fa65eec44e01a3aeb98a6b5d33238068fd8b5c4356111b9ef3de2e1a" exitCode=6 Jul 17 18:07:24.754434 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:24.753987 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" event={"ID":"6d7c2cba-034d-412b-8396-61aed8d1d9bb","Type":"ContainerDied","Data":"76482105fa65eec44e01a3aeb98a6b5d33238068fd8b5c4356111b9ef3de2e1a"} Jul 17 18:07:24.754434 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:24.754047 2573 scope.go:117] "RemoveContainer" containerID="1227937bbd40c6eccd56671471cbdaa55ca55576856c34b5c3578fe9df6fda74" Jul 17 18:07:24.754434 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:24.754322 2573 scope.go:117] "RemoveContainer" containerID="76482105fa65eec44e01a3aeb98a6b5d33238068fd8b5c4356111b9ef3de2e1a" Jul 17 18:07:24.754549 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:07:24.754529 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd_odh-ai-gateway-infra(6d7c2cba-034d-412b-8396-61aed8d1d9bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" podUID="6d7c2cba-034d-412b-8396-61aed8d1d9bb" Jul 17 18:07:24.772909 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:24.772884 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw"] Jul 17 18:07:24.787137 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:24.787104 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd"] Jul 17 18:07:24.952941 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:24.952911 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw" Jul 17 18:07:25.032906 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:25.032878 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6a7d4f3-87c2-46f4-82d0-eee80578ac6d-maas-api-service-ca\") pod \"c6a7d4f3-87c2-46f4-82d0-eee80578ac6d\" (UID: \"c6a7d4f3-87c2-46f4-82d0-eee80578ac6d\") " Jul 17 18:07:25.033245 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:25.033224 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a7d4f3-87c2-46f4-82d0-eee80578ac6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" (UID: "c6a7d4f3-87c2-46f4-82d0-eee80578ac6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:07:25.133792 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:25.133747 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6a7d4f3-87c2-46f4-82d0-eee80578ac6d-maas-api-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:07:25.762385 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:25.762355 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw" event={"ID":"c6a7d4f3-87c2-46f4-82d0-eee80578ac6d","Type":"ContainerDied","Data":"49de90f16050aafa2bcad5e728e60c82b276c17292b378dddae2f861e12398d6"} Jul 17 18:07:25.762385 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:25.762374 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw" Jul 17 18:07:25.762886 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:25.762411 2573 scope.go:117] "RemoveContainer" containerID="01604193cccecfa89ecf5fe713caa4bbbe6b56ec17f2de92a16641c9a147e5d8" Jul 17 18:07:25.793645 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:25.793602 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw"] Jul 17 18:07:25.799535 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:25.799508 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36649ebd-gd4sw"] Jul 17 18:07:25.935287 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:25.935263 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" Jul 17 18:07:26.042753 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:26.042662 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7c2cba-034d-412b-8396-61aed8d1d9bb-maas-api-service-ca\") pod \"6d7c2cba-034d-412b-8396-61aed8d1d9bb\" (UID: \"6d7c2cba-034d-412b-8396-61aed8d1d9bb\") " Jul 17 18:07:26.043043 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:26.043017 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7c2cba-034d-412b-8396-61aed8d1d9bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d7c2cba-034d-412b-8396-61aed8d1d9bb" (UID: "6d7c2cba-034d-412b-8396-61aed8d1d9bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:07:26.143736 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:26.143697 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7c2cba-034d-412b-8396-61aed8d1d9bb-maas-api-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:07:26.768829 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:26.768799 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" Jul 17 18:07:26.769265 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:26.768831 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd" event={"ID":"6d7c2cba-034d-412b-8396-61aed8d1d9bb","Type":"ContainerDied","Data":"b0d0c2640e83c4c102900b4ad73070dd03cf135eec47c294ed48e9176358d072"} Jul 17 18:07:26.769265 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:26.768899 2573 scope.go:117] "RemoveContainer" containerID="76482105fa65eec44e01a3aeb98a6b5d33238068fd8b5c4356111b9ef3de2e1a" Jul 17 18:07:26.791205 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:26.791179 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd"] Jul 17 18:07:26.797261 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:26.797235 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59516a0f-2cmdd"] Jul 17 18:07:27.778220 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:27.778185 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d7c2cba-034d-412b-8396-61aed8d1d9bb" path="/var/lib/kubelet/pods/6d7c2cba-034d-412b-8396-61aed8d1d9bb/volumes" Jul 17 18:07:27.778593 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:27.778539 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" path="/var/lib/kubelet/pods/c6a7d4f3-87c2-46f4-82d0-eee80578ac6d/volumes" Jul 17 18:07:28.743492 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:28.743462 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:28.843749 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:28.843712 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:29.788160 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:29.788126 2573 generic.go:356] "Generic (PLEG): container finished" podID="6f73b8a6-4553-4687-9149-e466ba680e15" containerID="2874f07204e95204af0d963f56e4de05053cafd1d55b2e66f6e26d68b6b5a6f2" exitCode=0 Jul 17 18:07:29.788405 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:29.788195 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" event={"ID":"6f73b8a6-4553-4687-9149-e466ba680e15","Type":"ContainerDied","Data":"2874f07204e95204af0d963f56e4de05053cafd1d55b2e66f6e26d68b6b5a6f2"} Jul 17 18:07:29.788658 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:29.788640 2573 scope.go:117] "RemoveContainer" containerID="2874f07204e95204af0d963f56e4de05053cafd1d55b2e66f6e26d68b6b5a6f2" Jul 17 18:07:29.789587 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:29.789562 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:07:29.789661 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:29.789641 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:07:29.789702 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:29.789687 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:07:30.698364 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:30.698331 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:30.795557 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:30.795520 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" event={"ID":"6f73b8a6-4553-4687-9149-e466ba680e15","Type":"ContainerStarted","Data":"71563ab977ae731f19936e18ce0aa4f3453d7f9e2a37a223f12b4f3029567f69"} Jul 17 18:07:31.012269 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:31.012225 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:31.013390 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:31.013354 2573 patch_prober.go:28] interesting pod/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 17 18:07:31.013538 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:31.013415 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" podUID="6f73b8a6-4553-4687-9149-e466ba680e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 17 18:07:32.012152 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:32.012111 2573 patch_prober.go:28] interesting pod/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 17 18:07:32.012750 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:32.012714 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" podUID="6f73b8a6-4553-4687-9149-e466ba680e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 17 18:07:33.012007 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:33.011972 2573 patch_prober.go:28] interesting pod/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 17 18:07:33.012727 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:33.012286 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" podUID="6f73b8a6-4553-4687-9149-e466ba680e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 17 18:07:34.012572 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:34.012531 2573 patch_prober.go:28] interesting pod/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 17 18:07:34.012889 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:34.012856 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" podUID="6f73b8a6-4553-4687-9149-e466ba680e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 17 18:07:35.015335 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:35.015302 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:35.015801 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:35.015594 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:35.016370 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:35.016352 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:07:40.039886 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.039859 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj"] Jul 17 18:07:40.040214 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.040101 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" podUID="bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6" containerName="maas-api" containerID="cri-o://19bc7812a8c6de0c8de03b74e9d171dbce704ece21f7ac86d5ab47e558c0c16c" gracePeriod=30 Jul 17 18:07:40.086356 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.086324 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:40.287495 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.287470 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" Jul 17 18:07:40.388450 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.388410 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-maas-api-tls\") pod \"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6\" (UID: \"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6\") " Jul 17 18:07:40.388683 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.388534 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxhsb\" (UniqueName: \"kubernetes.io/projected/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-kube-api-access-qxhsb\") pod \"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6\" (UID: \"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6\") " Jul 17 18:07:40.388683 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.388665 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-openshift-service-ca\") pod \"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6\" (UID: \"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6\") " Jul 17 18:07:40.389022 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.388998 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6" (UID: "bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:07:40.390521 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.390489 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6" (UID: "bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:07:40.390634 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.390577 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-kube-api-access-qxhsb" (OuterVolumeSpecName: "kube-api-access-qxhsb") pod "bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6" (UID: "bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6"). InnerVolumeSpecName "kube-api-access-qxhsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:07:40.490352 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.490313 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxhsb\" (UniqueName: \"kubernetes.io/projected/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-kube-api-access-qxhsb\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:07:40.490352 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.490344 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:07:40.490352 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.490354 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:07:40.858804 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.858765 2573 generic.go:356] "Generic (PLEG): container finished" podID="bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6" containerID="19bc7812a8c6de0c8de03b74e9d171dbce704ece21f7ac86d5ab47e558c0c16c" exitCode=0 Jul 17 18:07:40.859008 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.858814 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" event={"ID":"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6","Type":"ContainerDied","Data":"19bc7812a8c6de0c8de03b74e9d171dbce704ece21f7ac86d5ab47e558c0c16c"} Jul 17 18:07:40.859008 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.858821 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" Jul 17 18:07:40.859008 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.858865 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj" event={"ID":"bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6","Type":"ContainerDied","Data":"122ffb396665f8fdca79f9ab1bf5d0780c1495f54a59e98f04fd9cd4e5dd9c1b"} Jul 17 18:07:40.859008 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.858895 2573 scope.go:117] "RemoveContainer" containerID="19bc7812a8c6de0c8de03b74e9d171dbce704ece21f7ac86d5ab47e558c0c16c" Jul 17 18:07:40.868721 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.868697 2573 scope.go:117] "RemoveContainer" containerID="19bc7812a8c6de0c8de03b74e9d171dbce704ece21f7ac86d5ab47e558c0c16c" Jul 17 18:07:40.868984 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:07:40.868968 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19bc7812a8c6de0c8de03b74e9d171dbce704ece21f7ac86d5ab47e558c0c16c\": container with ID starting with 19bc7812a8c6de0c8de03b74e9d171dbce704ece21f7ac86d5ab47e558c0c16c not found: ID does not exist" containerID="19bc7812a8c6de0c8de03b74e9d171dbce704ece21f7ac86d5ab47e558c0c16c" Jul 17 18:07:40.869025 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.868993 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19bc7812a8c6de0c8de03b74e9d171dbce704ece21f7ac86d5ab47e558c0c16c"} err="failed to get container status \"19bc7812a8c6de0c8de03b74e9d171dbce704ece21f7ac86d5ab47e558c0c16c\": rpc error: code = NotFound desc = could not find container \"19bc7812a8c6de0c8de03b74e9d171dbce704ece21f7ac86d5ab47e558c0c16c\": container with ID starting with 19bc7812a8c6de0c8de03b74e9d171dbce704ece21f7ac86d5ab47e558c0c16c not found: ID does not exist" Jul 17 18:07:40.882009 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.881987 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj"] Jul 17 18:07:40.886415 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:40.886394 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f47fb1c3-864f9b94fd-rndxj"] Jul 17 18:07:41.777695 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:41.777659 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6" path="/var/lib/kubelet/pods/bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6/volumes" Jul 17 18:07:44.242673 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:44.242638 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:50.279322 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:50.279279 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:50.381841 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:50.381808 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:54.896528 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:54.896497 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246"] Jul 17 18:07:54.897066 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:54.896744 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" podUID="6f73b8a6-4553-4687-9149-e466ba680e15" containerName="istio-proxy" containerID="cri-o://71563ab977ae731f19936e18ce0aa4f3453d7f9e2a37a223f12b4f3029567f69" gracePeriod=30 Jul 17 18:07:54.922171 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:54.922135 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:55.153320 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:55.153230 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:55.281435 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:55.281405 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:55.378729 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:55.378689 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:55.495197 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:07:55.495165 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:00.150602 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.150573 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:08:00.288957 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.288924 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f73b8a6-4553-4687-9149-e466ba680e15-istio-token\") pod \"6f73b8a6-4553-4687-9149-e466ba680e15\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " Jul 17 18:08:00.288957 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.288962 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-credential-socket\") pod \"6f73b8a6-4553-4687-9149-e466ba680e15\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " Jul 17 18:08:00.289199 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.288998 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f73b8a6-4553-4687-9149-e466ba680e15-istiod-ca-cert\") pod \"6f73b8a6-4553-4687-9149-e466ba680e15\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " Jul 17 18:08:00.289199 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.289057 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-istio-data\") pod \"6f73b8a6-4553-4687-9149-e466ba680e15\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " Jul 17 18:08:00.289199 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.289091 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-istio-envoy\") pod \"6f73b8a6-4553-4687-9149-e466ba680e15\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " Jul 17 18:08:00.289199 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.289124 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-workload-socket\") pod \"6f73b8a6-4553-4687-9149-e466ba680e15\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " Jul 17 18:08:00.289199 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.289178 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpvfl\" (UniqueName: \"kubernetes.io/projected/6f73b8a6-4553-4687-9149-e466ba680e15-kube-api-access-hpvfl\") pod \"6f73b8a6-4553-4687-9149-e466ba680e15\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " Jul 17 18:08:00.289458 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.289208 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-workload-certs\") pod \"6f73b8a6-4553-4687-9149-e466ba680e15\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " Jul 17 18:08:00.289458 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.289236 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f73b8a6-4553-4687-9149-e466ba680e15-istio-podinfo\") pod \"6f73b8a6-4553-4687-9149-e466ba680e15\" (UID: \"6f73b8a6-4553-4687-9149-e466ba680e15\") " Jul 17 18:08:00.289458 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.289233 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f73b8a6-4553-4687-9149-e466ba680e15" (UID: "6f73b8a6-4553-4687-9149-e466ba680e15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:08:00.289604 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.289536 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f73b8a6-4553-4687-9149-e466ba680e15" (UID: "6f73b8a6-4553-4687-9149-e466ba680e15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:08:00.289604 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.289583 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f73b8a6-4553-4687-9149-e466ba680e15-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f73b8a6-4553-4687-9149-e466ba680e15" (UID: "6f73b8a6-4553-4687-9149-e466ba680e15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:08:00.289742 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.289700 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f73b8a6-4553-4687-9149-e466ba680e15" (UID: "6f73b8a6-4553-4687-9149-e466ba680e15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:08:00.289742 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.289709 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:08:00.289742 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.289731 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f73b8a6-4553-4687-9149-e466ba680e15-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:08:00.289885 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.289744 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:08:00.290148 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.290124 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f73b8a6-4553-4687-9149-e466ba680e15" (UID: "6f73b8a6-4553-4687-9149-e466ba680e15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:08:00.291288 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.291264 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f73b8a6-4553-4687-9149-e466ba680e15-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f73b8a6-4553-4687-9149-e466ba680e15" (UID: "6f73b8a6-4553-4687-9149-e466ba680e15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:08:00.291376 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.291346 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f73b8a6-4553-4687-9149-e466ba680e15-kube-api-access-hpvfl" (OuterVolumeSpecName: "kube-api-access-hpvfl") pod "6f73b8a6-4553-4687-9149-e466ba680e15" (UID: "6f73b8a6-4553-4687-9149-e466ba680e15"). InnerVolumeSpecName "kube-api-access-hpvfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:08:00.291654 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.291637 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f73b8a6-4553-4687-9149-e466ba680e15" (UID: "6f73b8a6-4553-4687-9149-e466ba680e15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:08:00.291723 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.291662 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f73b8a6-4553-4687-9149-e466ba680e15-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f73b8a6-4553-4687-9149-e466ba680e15" (UID: "6f73b8a6-4553-4687-9149-e466ba680e15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 18:08:00.390589 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.390550 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f73b8a6-4553-4687-9149-e466ba680e15-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:08:00.390589 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.390581 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:08:00.390589 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.390593 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:08:00.390850 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.390604 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f73b8a6-4553-4687-9149-e466ba680e15-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:08:00.390850 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.390641 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpvfl\" (UniqueName: \"kubernetes.io/projected/6f73b8a6-4553-4687-9149-e466ba680e15-kube-api-access-hpvfl\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:08:00.390850 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.390655 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f73b8a6-4553-4687-9149-e466ba680e15-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:08:00.958841 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.958757 2573 generic.go:356] "Generic (PLEG): container finished" podID="6f73b8a6-4553-4687-9149-e466ba680e15" containerID="71563ab977ae731f19936e18ce0aa4f3453d7f9e2a37a223f12b4f3029567f69" exitCode=0 Jul 17 18:08:00.958841 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.958812 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" Jul 17 18:08:00.959037 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.958842 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" event={"ID":"6f73b8a6-4553-4687-9149-e466ba680e15","Type":"ContainerDied","Data":"71563ab977ae731f19936e18ce0aa4f3453d7f9e2a37a223f12b4f3029567f69"} Jul 17 18:08:00.959037 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.958898 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246" event={"ID":"6f73b8a6-4553-4687-9149-e466ba680e15","Type":"ContainerDied","Data":"478416c3d4f528f551d11fbe31418825930a608514a136ab602867b8a45a0adb"} Jul 17 18:08:00.959037 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.958924 2573 scope.go:117] "RemoveContainer" containerID="71563ab977ae731f19936e18ce0aa4f3453d7f9e2a37a223f12b4f3029567f69" Jul 17 18:08:00.968437 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.968420 2573 scope.go:117] "RemoveContainer" containerID="2874f07204e95204af0d963f56e4de05053cafd1d55b2e66f6e26d68b6b5a6f2" Jul 17 18:08:00.976554 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.976529 2573 scope.go:117] "RemoveContainer" containerID="71563ab977ae731f19936e18ce0aa4f3453d7f9e2a37a223f12b4f3029567f69" Jul 17 18:08:00.976823 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:08:00.976804 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71563ab977ae731f19936e18ce0aa4f3453d7f9e2a37a223f12b4f3029567f69\": container with ID starting with 71563ab977ae731f19936e18ce0aa4f3453d7f9e2a37a223f12b4f3029567f69 not found: ID does not exist" containerID="71563ab977ae731f19936e18ce0aa4f3453d7f9e2a37a223f12b4f3029567f69" Jul 17 18:08:00.976880 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.976832 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71563ab977ae731f19936e18ce0aa4f3453d7f9e2a37a223f12b4f3029567f69"} err="failed to get container status \"71563ab977ae731f19936e18ce0aa4f3453d7f9e2a37a223f12b4f3029567f69\": rpc error: code = NotFound desc = could not find container \"71563ab977ae731f19936e18ce0aa4f3453d7f9e2a37a223f12b4f3029567f69\": container with ID starting with 71563ab977ae731f19936e18ce0aa4f3453d7f9e2a37a223f12b4f3029567f69 not found: ID does not exist" Jul 17 18:08:00.976880 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.976850 2573 scope.go:117] "RemoveContainer" containerID="2874f07204e95204af0d963f56e4de05053cafd1d55b2e66f6e26d68b6b5a6f2" Jul 17 18:08:00.977088 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:08:00.977071 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2874f07204e95204af0d963f56e4de05053cafd1d55b2e66f6e26d68b6b5a6f2\": container with ID starting with 2874f07204e95204af0d963f56e4de05053cafd1d55b2e66f6e26d68b6b5a6f2 not found: ID does not exist" containerID="2874f07204e95204af0d963f56e4de05053cafd1d55b2e66f6e26d68b6b5a6f2" Jul 17 18:08:00.977128 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.977094 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2874f07204e95204af0d963f56e4de05053cafd1d55b2e66f6e26d68b6b5a6f2"} err="failed to get container status \"2874f07204e95204af0d963f56e4de05053cafd1d55b2e66f6e26d68b6b5a6f2\": rpc error: code = NotFound desc = could not find container \"2874f07204e95204af0d963f56e4de05053cafd1d55b2e66f6e26d68b6b5a6f2\": container with ID starting with 2874f07204e95204af0d963f56e4de05053cafd1d55b2e66f6e26d68b6b5a6f2 not found: ID does not exist" Jul 17 18:08:00.983433 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.983409 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246"] Jul 17 18:08:00.991581 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:00.991555 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f47fb1c3-openshift-default-74d6f6cbc5-l6246"] Jul 17 18:08:01.778009 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:01.777974 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f73b8a6-4553-4687-9149-e466ba680e15" path="/var/lib/kubelet/pods/6f73b8a6-4553-4687-9149-e466ba680e15/volumes" Jul 17 18:08:01.905571 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:01.905527 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:02.015392 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:02.015349 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:02.104838 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:02.104733 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:02.249965 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:02.249932 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:08.352873 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:08.352836 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:08.427379 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:08.427347 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:15.146294 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.146251 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:15.433527 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.433439 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj"] Jul 17 18:08:15.433974 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.433955 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" containerName="revoke-keys" Jul 17 18:08:15.434078 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.433976 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" containerName="revoke-keys" Jul 17 18:08:15.434078 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.433994 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" containerName="revoke-keys" Jul 17 18:08:15.434078 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434002 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" containerName="revoke-keys" Jul 17 18:08:15.434078 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434025 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d7c2cba-034d-412b-8396-61aed8d1d9bb" containerName="revoke-keys" Jul 17 18:08:15.434078 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434034 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d7c2cba-034d-412b-8396-61aed8d1d9bb" containerName="revoke-keys" Jul 17 18:08:15.434078 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434048 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f73b8a6-4553-4687-9149-e466ba680e15" containerName="istio-proxy" Jul 17 18:08:15.434078 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434056 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f73b8a6-4553-4687-9149-e466ba680e15" containerName="istio-proxy" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434080 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6" containerName="maas-api" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434090 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6" containerName="maas-api" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434103 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d7c2cba-034d-412b-8396-61aed8d1d9bb" containerName="revoke-keys" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434111 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d7c2cba-034d-412b-8396-61aed8d1d9bb" containerName="revoke-keys" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434122 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d7c2cba-034d-412b-8396-61aed8d1d9bb" containerName="revoke-keys" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434131 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d7c2cba-034d-412b-8396-61aed8d1d9bb" containerName="revoke-keys" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434147 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f73b8a6-4553-4687-9149-e466ba680e15" containerName="istio-proxy" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434156 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f73b8a6-4553-4687-9149-e466ba680e15" containerName="istio-proxy" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434258 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" containerName="revoke-keys" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434273 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f73b8a6-4553-4687-9149-e466ba680e15" containerName="istio-proxy" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434281 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" containerName="revoke-keys" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434290 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d7c2cba-034d-412b-8396-61aed8d1d9bb" containerName="revoke-keys" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434302 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcc45eed-8dbb-4d90-9e03-aaaf2ce2abb6" containerName="maas-api" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434311 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d7c2cba-034d-412b-8396-61aed8d1d9bb" containerName="revoke-keys" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434322 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f73b8a6-4553-4687-9149-e466ba680e15" containerName="istio-proxy" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434332 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" containerName="revoke-keys" Jul 17 18:08:15.434422 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.434341 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d7c2cba-034d-412b-8396-61aed8d1d9bb" containerName="revoke-keys" Jul 17 18:08:15.437984 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.437961 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.441499 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.441476 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a60650e8-openshift-default-dockercfg-28l9g\"" Jul 17 18:08:15.458103 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.458076 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj"] Jul 17 18:08:15.473003 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.472970 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:15.544765 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.544733 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-workload-certs\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.544925 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.544782 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-credential-socket\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.544925 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.544812 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070a1565-cfc6-43bd-b518-63c348f30745-istio-token\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.544925 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.544853 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070a1565-cfc6-43bd-b518-63c348f30745-istio-podinfo\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.545031 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.544928 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070a1565-cfc6-43bd-b518-63c348f30745-istiod-ca-cert\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.545031 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.544954 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngzm8\" (UniqueName: \"kubernetes.io/projected/070a1565-cfc6-43bd-b518-63c348f30745-kube-api-access-ngzm8\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.545031 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.544980 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-workload-socket\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.545031 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.545000 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-istio-data\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.545153 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.545050 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-istio-envoy\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.646297 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.646260 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070a1565-cfc6-43bd-b518-63c348f30745-istio-podinfo\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.646468 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.646315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070a1565-cfc6-43bd-b518-63c348f30745-istiod-ca-cert\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.646468 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.646333 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngzm8\" (UniqueName: \"kubernetes.io/projected/070a1565-cfc6-43bd-b518-63c348f30745-kube-api-access-ngzm8\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.646468 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.646368 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-workload-socket\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.646661 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.646520 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-istio-data\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.646661 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.646581 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-istio-envoy\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.646768 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.646663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-workload-certs\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.646768 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.646742 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-credential-socket\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.646875 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.646781 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070a1565-cfc6-43bd-b518-63c348f30745-istio-token\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.646875 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.646798 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-workload-socket\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.646984 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.646891 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-istio-data\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.647075 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.647048 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-workload-certs\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.647164 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.647119 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-credential-socket\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.647223 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.647174 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070a1565-cfc6-43bd-b518-63c348f30745-istiod-ca-cert\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.648869 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.648847 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-istio-envoy\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.649098 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.649077 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070a1565-cfc6-43bd-b518-63c348f30745-istio-podinfo\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.655346 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.655324 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070a1565-cfc6-43bd-b518-63c348f30745-istio-token\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.655582 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.655560 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngzm8\" (UniqueName: \"kubernetes.io/projected/070a1565-cfc6-43bd-b518-63c348f30745-kube-api-access-ngzm8\") pod \"e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.748306 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.748270 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:15.785432 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.785356 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:15.919860 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.919835 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj"] Jul 17 18:08:15.921825 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:08:15.921798 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070a1565_cfc6_43bd_b518_63c348f30745.slice/crio-bbf777006ba8252b3907c9425b2fbea0500003c59a80539a2d3806b092a10fef WatchSource:0}: Error finding container bbf777006ba8252b3907c9425b2fbea0500003c59a80539a2d3806b092a10fef: Status 404 returned error can't find the container with id bbf777006ba8252b3907c9425b2fbea0500003c59a80539a2d3806b092a10fef Jul 17 18:08:15.923963 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.923933 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:08:15.924035 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.923992 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:08:15.924035 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:15.924020 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:08:16.039601 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:16.039514 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" event={"ID":"070a1565-cfc6-43bd-b518-63c348f30745","Type":"ContainerStarted","Data":"cdecdf97b71f827e063ea9b1e4cfea698642329f1250f45356aa8208a1651670"} Jul 17 18:08:16.039601 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:16.039567 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" event={"ID":"070a1565-cfc6-43bd-b518-63c348f30745","Type":"ContainerStarted","Data":"bbf777006ba8252b3907c9425b2fbea0500003c59a80539a2d3806b092a10fef"} Jul 17 18:08:16.094797 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:16.094726 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" podStartSLOduration=1.094703148 podStartE2EDuration="1.094703148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:08:16.093014816 +0000 UTC m=+2616.828438867" watchObservedRunningTime="2026-07-17 18:08:16.094703148 +0000 UTC m=+2616.830127198" Jul 17 18:08:16.749122 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:16.749076 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:16.750759 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:16.750727 2573 patch_prober.go:28] interesting pod/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 17 18:08:16.750845 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:16.750792 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" podUID="070a1565-cfc6-43bd-b518-63c348f30745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 17 18:08:17.752849 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:17.752818 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:18.049510 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.049423 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:18.050404 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.050383 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:18.132967 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.132932 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:18.210019 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.209983 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql"] Jul 17 18:08:18.210412 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.210399 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" containerName="revoke-keys" Jul 17 18:08:18.210466 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.210416 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a7d4f3-87c2-46f4-82d0-eee80578ac6d" containerName="revoke-keys" Jul 17 18:08:18.213479 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.213460 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" Jul 17 18:08:18.217022 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.217001 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a60650e8-serving-cert\"" Jul 17 18:08:18.219993 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.219969 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:18.230204 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.230172 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql"] Jul 17 18:08:18.273591 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.273548 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c3d10d-61bc-4761-9005-bee1a33d97d1-maas-api-tls\") pod \"maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql\" (UID: \"55c3d10d-61bc-4761-9005-bee1a33d97d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" Jul 17 18:08:18.273778 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.273596 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c3d10d-61bc-4761-9005-bee1a33d97d1-openshift-service-ca\") pod \"maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql\" (UID: \"55c3d10d-61bc-4761-9005-bee1a33d97d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" Jul 17 18:08:18.273778 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.273707 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzgq5\" (UniqueName: \"kubernetes.io/projected/55c3d10d-61bc-4761-9005-bee1a33d97d1-kube-api-access-lzgq5\") pod \"maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql\" (UID: \"55c3d10d-61bc-4761-9005-bee1a33d97d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" Jul 17 18:08:18.302478 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.302398 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:18.375336 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.375287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c3d10d-61bc-4761-9005-bee1a33d97d1-maas-api-tls\") pod \"maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql\" (UID: \"55c3d10d-61bc-4761-9005-bee1a33d97d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" Jul 17 18:08:18.375530 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.375345 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c3d10d-61bc-4761-9005-bee1a33d97d1-openshift-service-ca\") pod \"maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql\" (UID: \"55c3d10d-61bc-4761-9005-bee1a33d97d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" Jul 17 18:08:18.375530 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.375407 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzgq5\" (UniqueName: \"kubernetes.io/projected/55c3d10d-61bc-4761-9005-bee1a33d97d1-kube-api-access-lzgq5\") pod \"maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql\" (UID: \"55c3d10d-61bc-4761-9005-bee1a33d97d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" Jul 17 18:08:18.376206 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.376177 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c3d10d-61bc-4761-9005-bee1a33d97d1-openshift-service-ca\") pod \"maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql\" (UID: \"55c3d10d-61bc-4761-9005-bee1a33d97d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" Jul 17 18:08:18.377853 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.377830 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c3d10d-61bc-4761-9005-bee1a33d97d1-maas-api-tls\") pod \"maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql\" (UID: \"55c3d10d-61bc-4761-9005-bee1a33d97d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" Jul 17 18:08:18.384470 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.384438 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzgq5\" (UniqueName: \"kubernetes.io/projected/55c3d10d-61bc-4761-9005-bee1a33d97d1-kube-api-access-lzgq5\") pod \"maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql\" (UID: \"55c3d10d-61bc-4761-9005-bee1a33d97d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" Jul 17 18:08:18.403261 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.403231 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:18.524523 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.524490 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" Jul 17 18:08:18.701760 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.701737 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql"] Jul 17 18:08:18.703637 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:08:18.703596 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c3d10d_61bc_4761_9005_bee1a33d97d1.slice/crio-103ad4547949731e2e69b1764ea85c635fdd02d058883779d6610ef928a59f25 WatchSource:0}: Error finding container 103ad4547949731e2e69b1764ea85c635fdd02d058883779d6610ef928a59f25: Status 404 returned error can't find the container with id 103ad4547949731e2e69b1764ea85c635fdd02d058883779d6610ef928a59f25 Jul 17 18:08:18.963421 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:18.963334 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:19.055068 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:19.055035 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" event={"ID":"55c3d10d-61bc-4761-9005-bee1a33d97d1","Type":"ContainerStarted","Data":"103ad4547949731e2e69b1764ea85c635fdd02d058883779d6610ef928a59f25"} Jul 17 18:08:19.064355 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:19.064331 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:20.061754 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:20.061709 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" event={"ID":"55c3d10d-61bc-4761-9005-bee1a33d97d1","Type":"ContainerStarted","Data":"e34d5271161c375e43b4dae888130251e3151b2249d6062adae8d85d98ba6b7f"} Jul 17 18:08:20.087182 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:20.087122 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" podStartSLOduration=1.6706383420000002 podStartE2EDuration="2.087106641s" totalImagesPullingTime="416.468299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:08:20.086294164 +0000 UTC m=+2620.821718214" watchObservedRunningTime="2026-07-17 18:08:20.087106641 +0000 UTC m=+2620.822530690" Jul 17 18:08:21.066183 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:21.066152 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" Jul 17 18:08:22.525277 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:22.525240 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:27.077687 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:27.077657 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" Jul 17 18:08:27.527533 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:27.527502 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:30.114586 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:30.114558 2573 generic.go:356] "Generic (PLEG): container finished" podID="070a1565-cfc6-43bd-b518-63c348f30745" containerID="cdecdf97b71f827e063ea9b1e4cfea698642329f1250f45356aa8208a1651670" exitCode=0 Jul 17 18:08:30.115035 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:30.114651 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" event={"ID":"070a1565-cfc6-43bd-b518-63c348f30745","Type":"ContainerDied","Data":"cdecdf97b71f827e063ea9b1e4cfea698642329f1250f45356aa8208a1651670"} Jul 17 18:08:30.115079 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:30.115066 2573 scope.go:117] "RemoveContainer" containerID="cdecdf97b71f827e063ea9b1e4cfea698642329f1250f45356aa8208a1651670" Jul 17 18:08:30.116056 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:30.116028 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:08:30.116117 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:30.116096 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:08:30.116152 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:30.116126 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:08:31.125638 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:31.125576 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" event={"ID":"070a1565-cfc6-43bd-b518-63c348f30745","Type":"ContainerStarted","Data":"a6b20a4050a1ecd8eeee2f75122d9dba0c966bb497b735f1ede0dd0db98a75ee"} Jul 17 18:08:31.748717 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:31.748664 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:31.749736 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:31.749706 2573 patch_prober.go:28] interesting pod/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 17 18:08:31.749881 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:31.749767 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" podUID="070a1565-cfc6-43bd-b518-63c348f30745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 17 18:08:32.749242 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:32.749204 2573 patch_prober.go:28] interesting pod/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 17 18:08:32.749683 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:32.749265 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" podUID="070a1565-cfc6-43bd-b518-63c348f30745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 17 18:08:33.050228 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:33.050139 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:33.748759 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:33.748719 2573 patch_prober.go:28] interesting pod/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 17 18:08:33.748956 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:33.748776 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" podUID="070a1565-cfc6-43bd-b518-63c348f30745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 17 18:08:34.748815 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:34.748778 2573 patch_prober.go:28] interesting pod/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 17 18:08:34.749235 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:34.748833 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" podUID="070a1565-cfc6-43bd-b518-63c348f30745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 17 18:08:35.752458 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:35.752429 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:08:35.753586 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:08:35.753564 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:09:06.306824 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:06.306791 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql"] Jul 17 18:09:06.307379 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:06.307018 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" podUID="55c3d10d-61bc-4761-9005-bee1a33d97d1" containerName="maas-api" containerID="cri-o://e34d5271161c375e43b4dae888130251e3151b2249d6062adae8d85d98ba6b7f" gracePeriod=30 Jul 17 18:09:06.339858 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:06.339832 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:06.557156 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:06.557086 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" Jul 17 18:09:06.661406 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:06.661374 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c3d10d-61bc-4761-9005-bee1a33d97d1-openshift-service-ca\") pod \"55c3d10d-61bc-4761-9005-bee1a33d97d1\" (UID: \"55c3d10d-61bc-4761-9005-bee1a33d97d1\") " Jul 17 18:09:06.661598 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:06.661425 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzgq5\" (UniqueName: \"kubernetes.io/projected/55c3d10d-61bc-4761-9005-bee1a33d97d1-kube-api-access-lzgq5\") pod \"55c3d10d-61bc-4761-9005-bee1a33d97d1\" (UID: \"55c3d10d-61bc-4761-9005-bee1a33d97d1\") " Jul 17 18:09:06.661598 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:06.661576 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c3d10d-61bc-4761-9005-bee1a33d97d1-maas-api-tls\") pod \"55c3d10d-61bc-4761-9005-bee1a33d97d1\" (UID: \"55c3d10d-61bc-4761-9005-bee1a33d97d1\") " Jul 17 18:09:06.661853 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:06.661827 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c3d10d-61bc-4761-9005-bee1a33d97d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "55c3d10d-61bc-4761-9005-bee1a33d97d1" (UID: "55c3d10d-61bc-4761-9005-bee1a33d97d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:09:06.661939 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:06.661916 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c3d10d-61bc-4761-9005-bee1a33d97d1-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:09:06.663507 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:06.663483 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c3d10d-61bc-4761-9005-bee1a33d97d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "55c3d10d-61bc-4761-9005-bee1a33d97d1" (UID: "55c3d10d-61bc-4761-9005-bee1a33d97d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:09:06.663598 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:06.663584 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c3d10d-61bc-4761-9005-bee1a33d97d1-kube-api-access-lzgq5" (OuterVolumeSpecName: "kube-api-access-lzgq5") pod "55c3d10d-61bc-4761-9005-bee1a33d97d1" (UID: "55c3d10d-61bc-4761-9005-bee1a33d97d1"). InnerVolumeSpecName "kube-api-access-lzgq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:09:06.763289 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:06.763251 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c3d10d-61bc-4761-9005-bee1a33d97d1-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:09:06.763289 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:06.763284 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzgq5\" (UniqueName: \"kubernetes.io/projected/55c3d10d-61bc-4761-9005-bee1a33d97d1-kube-api-access-lzgq5\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:09:07.311066 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:07.311026 2573 generic.go:356] "Generic (PLEG): container finished" podID="55c3d10d-61bc-4761-9005-bee1a33d97d1" containerID="e34d5271161c375e43b4dae888130251e3151b2249d6062adae8d85d98ba6b7f" exitCode=0 Jul 17 18:09:07.311066 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:07.311068 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" Jul 17 18:09:07.311537 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:07.311107 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" event={"ID":"55c3d10d-61bc-4761-9005-bee1a33d97d1","Type":"ContainerDied","Data":"e34d5271161c375e43b4dae888130251e3151b2249d6062adae8d85d98ba6b7f"} Jul 17 18:09:07.311537 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:07.311158 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql" event={"ID":"55c3d10d-61bc-4761-9005-bee1a33d97d1","Type":"ContainerDied","Data":"103ad4547949731e2e69b1764ea85c635fdd02d058883779d6610ef928a59f25"} Jul 17 18:09:07.311537 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:07.311179 2573 scope.go:117] "RemoveContainer" containerID="e34d5271161c375e43b4dae888130251e3151b2249d6062adae8d85d98ba6b7f" Jul 17 18:09:07.320644 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:07.320601 2573 scope.go:117] "RemoveContainer" containerID="e34d5271161c375e43b4dae888130251e3151b2249d6062adae8d85d98ba6b7f" Jul 17 18:09:07.320940 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:09:07.320920 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34d5271161c375e43b4dae888130251e3151b2249d6062adae8d85d98ba6b7f\": container with ID starting with e34d5271161c375e43b4dae888130251e3151b2249d6062adae8d85d98ba6b7f not found: ID does not exist" containerID="e34d5271161c375e43b4dae888130251e3151b2249d6062adae8d85d98ba6b7f" Jul 17 18:09:07.320990 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:07.320948 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34d5271161c375e43b4dae888130251e3151b2249d6062adae8d85d98ba6b7f"} err="failed to get container status \"e34d5271161c375e43b4dae888130251e3151b2249d6062adae8d85d98ba6b7f\": rpc error: code = NotFound desc = could not find container \"e34d5271161c375e43b4dae888130251e3151b2249d6062adae8d85d98ba6b7f\": container with ID starting with e34d5271161c375e43b4dae888130251e3151b2249d6062adae8d85d98ba6b7f not found: ID does not exist" Jul 17 18:09:07.334158 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:07.334133 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql"] Jul 17 18:09:07.339591 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:07.339567 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a60650e8-cd4c46c95-nkvql"] Jul 17 18:09:07.777438 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:07.777405 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55c3d10d-61bc-4761-9005-bee1a33d97d1" path="/var/lib/kubelet/pods/55c3d10d-61bc-4761-9005-bee1a33d97d1/volumes" Jul 17 18:09:10.754019 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:10.753976 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:15.830691 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:15.830653 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:26.467558 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:26.467516 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:26.565613 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:26.565578 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:39.916580 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:39.916546 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 18:09:39.919719 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:39.919699 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 18:09:51.881277 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:51.881237 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:53.313423 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:53.313391 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:59.329978 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:59.329940 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:59.428591 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:09:59.428556 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:02.957329 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:02.957294 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj"] Jul 17 18:10:02.957754 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:02.957555 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" podUID="070a1565-cfc6-43bd-b518-63c348f30745" containerName="istio-proxy" containerID="cri-o://a6b20a4050a1ecd8eeee2f75122d9dba0c966bb497b735f1ede0dd0db98a75ee" gracePeriod=30 Jul 17 18:10:02.983418 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:02.983374 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:03.218889 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:03.218799 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:04.330722 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:04.330693 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:04.426030 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:04.425997 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:04.649008 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:04.648920 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:05.259187 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:05.259156 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:05.354107 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:05.354075 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:05.454169 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:05.454136 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:05.755602 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:05.755570 2573 patch_prober.go:28] interesting pod/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 18:10:05.755778 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:05.755645 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" podUID="070a1565-cfc6-43bd-b518-63c348f30745" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 18:10:06.808298 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:06.808263 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:08.239813 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.239781 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:10:08.373692 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.373648 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070a1565-cfc6-43bd-b518-63c348f30745-istio-token\") pod \"070a1565-cfc6-43bd-b518-63c348f30745\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " Jul 17 18:10:08.373883 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.373713 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-istio-data\") pod \"070a1565-cfc6-43bd-b518-63c348f30745\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " Jul 17 18:10:08.373883 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.373734 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070a1565-cfc6-43bd-b518-63c348f30745-istio-podinfo\") pod \"070a1565-cfc6-43bd-b518-63c348f30745\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " Jul 17 18:10:08.373883 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.373757 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-credential-socket\") pod \"070a1565-cfc6-43bd-b518-63c348f30745\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " Jul 17 18:10:08.373883 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.373782 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngzm8\" (UniqueName: \"kubernetes.io/projected/070a1565-cfc6-43bd-b518-63c348f30745-kube-api-access-ngzm8\") pod \"070a1565-cfc6-43bd-b518-63c348f30745\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " Jul 17 18:10:08.373883 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.373867 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-istio-envoy\") pod \"070a1565-cfc6-43bd-b518-63c348f30745\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " Jul 17 18:10:08.374171 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.373900 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070a1565-cfc6-43bd-b518-63c348f30745-istiod-ca-cert\") pod \"070a1565-cfc6-43bd-b518-63c348f30745\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " Jul 17 18:10:08.374171 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.373966 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-workload-socket\") pod \"070a1565-cfc6-43bd-b518-63c348f30745\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " Jul 17 18:10:08.374171 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.373993 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-workload-certs\") pod \"070a1565-cfc6-43bd-b518-63c348f30745\" (UID: \"070a1565-cfc6-43bd-b518-63c348f30745\") " Jul 17 18:10:08.374844 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.374410 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070a1565-cfc6-43bd-b518-63c348f30745-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "070a1565-cfc6-43bd-b518-63c348f30745" (UID: "070a1565-cfc6-43bd-b518-63c348f30745"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:10:08.374844 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.374492 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "070a1565-cfc6-43bd-b518-63c348f30745" (UID: "070a1565-cfc6-43bd-b518-63c348f30745"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:10:08.374844 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.374508 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "070a1565-cfc6-43bd-b518-63c348f30745" (UID: "070a1565-cfc6-43bd-b518-63c348f30745"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:10:08.374844 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.374591 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "070a1565-cfc6-43bd-b518-63c348f30745" (UID: "070a1565-cfc6-43bd-b518-63c348f30745"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:10:08.374844 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.374659 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-istio-data" (OuterVolumeSpecName: "istio-data") pod "070a1565-cfc6-43bd-b518-63c348f30745" (UID: "070a1565-cfc6-43bd-b518-63c348f30745"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:10:08.376293 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.376266 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070a1565-cfc6-43bd-b518-63c348f30745-istio-token" (OuterVolumeSpecName: "istio-token") pod "070a1565-cfc6-43bd-b518-63c348f30745" (UID: "070a1565-cfc6-43bd-b518-63c348f30745"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:10:08.376728 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.376708 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "070a1565-cfc6-43bd-b518-63c348f30745" (UID: "070a1565-cfc6-43bd-b518-63c348f30745"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:10:08.376963 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.376935 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070a1565-cfc6-43bd-b518-63c348f30745-kube-api-access-ngzm8" (OuterVolumeSpecName: "kube-api-access-ngzm8") pod "070a1565-cfc6-43bd-b518-63c348f30745" (UID: "070a1565-cfc6-43bd-b518-63c348f30745"). InnerVolumeSpecName "kube-api-access-ngzm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:10:08.377048 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.376961 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/070a1565-cfc6-43bd-b518-63c348f30745-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "070a1565-cfc6-43bd-b518-63c348f30745" (UID: "070a1565-cfc6-43bd-b518-63c348f30745"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 18:10:08.475115 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.475037 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:10:08.475115 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.475094 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070a1565-cfc6-43bd-b518-63c348f30745-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:10:08.475115 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.475103 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:10:08.475115 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.475112 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:10:08.475365 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.475120 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070a1565-cfc6-43bd-b518-63c348f30745-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:10:08.475365 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.475128 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:10:08.475365 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.475135 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070a1565-cfc6-43bd-b518-63c348f30745-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:10:08.475365 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.475142 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070a1565-cfc6-43bd-b518-63c348f30745-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:10:08.475365 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.475151 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngzm8\" (UniqueName: \"kubernetes.io/projected/070a1565-cfc6-43bd-b518-63c348f30745-kube-api-access-ngzm8\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:10:08.626037 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.625998 2573 generic.go:356] "Generic (PLEG): container finished" podID="070a1565-cfc6-43bd-b518-63c348f30745" containerID="a6b20a4050a1ecd8eeee2f75122d9dba0c966bb497b735f1ede0dd0db98a75ee" exitCode=0 Jul 17 18:10:08.626234 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.626054 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" Jul 17 18:10:08.626234 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.626074 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" event={"ID":"070a1565-cfc6-43bd-b518-63c348f30745","Type":"ContainerDied","Data":"a6b20a4050a1ecd8eeee2f75122d9dba0c966bb497b735f1ede0dd0db98a75ee"} Jul 17 18:10:08.626234 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.626144 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj" event={"ID":"070a1565-cfc6-43bd-b518-63c348f30745","Type":"ContainerDied","Data":"bbf777006ba8252b3907c9425b2fbea0500003c59a80539a2d3806b092a10fef"} Jul 17 18:10:08.626234 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.626166 2573 scope.go:117] "RemoveContainer" containerID="a6b20a4050a1ecd8eeee2f75122d9dba0c966bb497b735f1ede0dd0db98a75ee" Jul 17 18:10:08.639233 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.639208 2573 scope.go:117] "RemoveContainer" containerID="cdecdf97b71f827e063ea9b1e4cfea698642329f1250f45356aa8208a1651670" Jul 17 18:10:08.654190 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.654156 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj"] Jul 17 18:10:08.656039 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.656022 2573 scope.go:117] "RemoveContainer" containerID="a6b20a4050a1ecd8eeee2f75122d9dba0c966bb497b735f1ede0dd0db98a75ee" Jul 17 18:10:08.656366 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:10:08.656339 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b20a4050a1ecd8eeee2f75122d9dba0c966bb497b735f1ede0dd0db98a75ee\": container with ID starting with a6b20a4050a1ecd8eeee2f75122d9dba0c966bb497b735f1ede0dd0db98a75ee not found: ID does not exist" containerID="a6b20a4050a1ecd8eeee2f75122d9dba0c966bb497b735f1ede0dd0db98a75ee" Jul 17 18:10:08.656419 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.656382 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b20a4050a1ecd8eeee2f75122d9dba0c966bb497b735f1ede0dd0db98a75ee"} err="failed to get container status \"a6b20a4050a1ecd8eeee2f75122d9dba0c966bb497b735f1ede0dd0db98a75ee\": rpc error: code = NotFound desc = could not find container \"a6b20a4050a1ecd8eeee2f75122d9dba0c966bb497b735f1ede0dd0db98a75ee\": container with ID starting with a6b20a4050a1ecd8eeee2f75122d9dba0c966bb497b735f1ede0dd0db98a75ee not found: ID does not exist" Jul 17 18:10:08.656419 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.656412 2573 scope.go:117] "RemoveContainer" containerID="cdecdf97b71f827e063ea9b1e4cfea698642329f1250f45356aa8208a1651670" Jul 17 18:10:08.656679 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:10:08.656659 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdecdf97b71f827e063ea9b1e4cfea698642329f1250f45356aa8208a1651670\": container with ID starting with cdecdf97b71f827e063ea9b1e4cfea698642329f1250f45356aa8208a1651670 not found: ID does not exist" containerID="cdecdf97b71f827e063ea9b1e4cfea698642329f1250f45356aa8208a1651670" Jul 17 18:10:08.656733 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.656692 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdecdf97b71f827e063ea9b1e4cfea698642329f1250f45356aa8208a1651670"} err="failed to get container status \"cdecdf97b71f827e063ea9b1e4cfea698642329f1250f45356aa8208a1651670\": rpc error: code = NotFound desc = could not find container \"cdecdf97b71f827e063ea9b1e4cfea698642329f1250f45356aa8208a1651670\": container with ID starting with cdecdf97b71f827e063ea9b1e4cfea698642329f1250f45356aa8208a1651670 not found: ID does not exist" Jul 17 18:10:08.664096 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:08.664074 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a60650e8-openshift-default-dc97d9cc5-sm9mj"] Jul 17 18:10:09.777527 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:09.777488 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="070a1565-cfc6-43bd-b518-63c348f30745" path="/var/lib/kubelet/pods/070a1565-cfc6-43bd-b518-63c348f30745/volumes" Jul 17 18:10:12.748884 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:12.748844 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:12.848081 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:12.848051 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:13.422927 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:13.422892 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:18.298829 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:18.298787 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:18.400441 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:18.400413 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:19.374297 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:19.374266 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:19.477554 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:19.477518 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:19.578383 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:19.578353 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:30.039235 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:30.039197 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:31.292432 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:31.292397 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:34.374749 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.374716 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt"] Jul 17 18:10:34.375198 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.375180 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c3d10d-61bc-4761-9005-bee1a33d97d1" containerName="maas-api" Jul 17 18:10:34.375246 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.375200 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c3d10d-61bc-4761-9005-bee1a33d97d1" containerName="maas-api" Jul 17 18:10:34.375246 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.375211 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="070a1565-cfc6-43bd-b518-63c348f30745" containerName="istio-proxy" Jul 17 18:10:34.375246 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.375217 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="070a1565-cfc6-43bd-b518-63c348f30745" containerName="istio-proxy" Jul 17 18:10:34.375246 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.375230 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="070a1565-cfc6-43bd-b518-63c348f30745" containerName="istio-proxy" Jul 17 18:10:34.375246 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.375236 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="070a1565-cfc6-43bd-b518-63c348f30745" containerName="istio-proxy" Jul 17 18:10:34.375401 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.375309 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="070a1565-cfc6-43bd-b518-63c348f30745" containerName="istio-proxy" Jul 17 18:10:34.375401 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.375318 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c3d10d-61bc-4761-9005-bee1a33d97d1" containerName="maas-api" Jul 17 18:10:34.375401 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.375325 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="070a1565-cfc6-43bd-b518-63c348f30745" containerName="istio-proxy" Jul 17 18:10:34.378583 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.378567 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt" Jul 17 18:10:34.381165 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.381143 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4f9n\"" Jul 17 18:10:34.385941 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.385917 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt"] Jul 17 18:10:34.428086 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.428055 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a911bd-67de-4890-8680-34f41499314c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt\" (UID: \"f6a911bd-67de-4890-8680-34f41499314c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt" Jul 17 18:10:34.529610 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.529568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a911bd-67de-4890-8680-34f41499314c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt\" (UID: \"f6a911bd-67de-4890-8680-34f41499314c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt" Jul 17 18:10:34.530212 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.530193 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a911bd-67de-4890-8680-34f41499314c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt\" (UID: \"f6a911bd-67de-4890-8680-34f41499314c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt" Jul 17 18:10:34.691313 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.691232 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt" Jul 17 18:10:34.821859 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:34.821834 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt"] Jul 17 18:10:34.823881 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:10:34.823857 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a911bd_67de_4890_8680_34f41499314c.slice/crio-15b3712647e87b8ad1116af6aef62931a6ca219f26ae52be5c3ee4cd9b51dc32 WatchSource:0}: Error finding container 15b3712647e87b8ad1116af6aef62931a6ca219f26ae52be5c3ee4cd9b51dc32: Status 404 returned error can't find the container with id 15b3712647e87b8ad1116af6aef62931a6ca219f26ae52be5c3ee4cd9b51dc32 Jul 17 18:10:35.760889 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:35.760855 2573 generic.go:356] "Generic (PLEG): container finished" podID="f6a911bd-67de-4890-8680-34f41499314c" containerID="9aa9cfb0664c2afa719f6725d23fb0d1f792180c8a2882cc7161aa3bd1a290b9" exitCode=6 Jul 17 18:10:35.761393 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:35.760939 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt" event={"ID":"f6a911bd-67de-4890-8680-34f41499314c","Type":"ContainerDied","Data":"9aa9cfb0664c2afa719f6725d23fb0d1f792180c8a2882cc7161aa3bd1a290b9"} Jul 17 18:10:35.761393 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:35.761045 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt" event={"ID":"f6a911bd-67de-4890-8680-34f41499314c","Type":"ContainerStarted","Data":"15b3712647e87b8ad1116af6aef62931a6ca219f26ae52be5c3ee4cd9b51dc32"} Jul 17 18:10:35.761393 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:35.761173 2573 scope.go:117] "RemoveContainer" containerID="9aa9cfb0664c2afa719f6725d23fb0d1f792180c8a2882cc7161aa3bd1a290b9" Jul 17 18:10:36.767675 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:36.767645 2573 generic.go:356] "Generic (PLEG): container finished" podID="f6a911bd-67de-4890-8680-34f41499314c" containerID="00a65ea7c751357c2783edcb96c07d0b413b72a551842f65eb2e568d995118bd" exitCode=6 Jul 17 18:10:36.768099 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:36.767694 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt" event={"ID":"f6a911bd-67de-4890-8680-34f41499314c","Type":"ContainerDied","Data":"00a65ea7c751357c2783edcb96c07d0b413b72a551842f65eb2e568d995118bd"} Jul 17 18:10:36.768099 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:36.767744 2573 scope.go:117] "RemoveContainer" containerID="9aa9cfb0664c2afa719f6725d23fb0d1f792180c8a2882cc7161aa3bd1a290b9" Jul 17 18:10:36.768099 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:36.767972 2573 scope.go:117] "RemoveContainer" containerID="00a65ea7c751357c2783edcb96c07d0b413b72a551842f65eb2e568d995118bd" Jul 17 18:10:36.768270 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:10:36.768192 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt_odh-ai-gateway-infra(f6a911bd-67de-4890-8680-34f41499314c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt" podUID="f6a911bd-67de-4890-8680-34f41499314c" Jul 17 18:10:37.774867 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:37.774838 2573 scope.go:117] "RemoveContainer" containerID="00a65ea7c751357c2783edcb96c07d0b413b72a551842f65eb2e568d995118bd" Jul 17 18:10:37.775247 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:10:37.775103 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt_odh-ai-gateway-infra(f6a911bd-67de-4890-8680-34f41499314c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt" podUID="f6a911bd-67de-4890-8680-34f41499314c" Jul 17 18:10:39.939826 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:39.939793 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc"] Jul 17 18:10:39.944661 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:39.944641 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" Jul 17 18:10:39.955982 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:39.954515 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc"] Jul 17 18:10:40.086857 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:40.086826 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9383a51b-695d-4616-a10c-742f6e95764d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc\" (UID: \"9383a51b-695d-4616-a10c-742f6e95764d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" Jul 17 18:10:40.188239 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:40.188201 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9383a51b-695d-4616-a10c-742f6e95764d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc\" (UID: \"9383a51b-695d-4616-a10c-742f6e95764d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" Jul 17 18:10:40.188881 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:40.188861 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9383a51b-695d-4616-a10c-742f6e95764d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc\" (UID: \"9383a51b-695d-4616-a10c-742f6e95764d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" Jul 17 18:10:40.261469 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:40.261434 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" Jul 17 18:10:40.394131 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:40.394096 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc"] Jul 17 18:10:40.397605 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:10:40.397576 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9383a51b_695d_4616_a10c_742f6e95764d.slice/crio-32db76b241b194d741de449ed291f3d27b05a25ade1f880158afb2c9f0830134 WatchSource:0}: Error finding container 32db76b241b194d741de449ed291f3d27b05a25ade1f880158afb2c9f0830134: Status 404 returned error can't find the container with id 32db76b241b194d741de449ed291f3d27b05a25ade1f880158afb2c9f0830134 Jul 17 18:10:40.790882 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:40.790843 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" event={"ID":"9383a51b-695d-4616-a10c-742f6e95764d","Type":"ContainerStarted","Data":"6626b491926b2798e3293a68aa6080bfe21f85df01995285e62463f207a7c8a1"} Jul 17 18:10:40.791063 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:40.790897 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" event={"ID":"9383a51b-695d-4616-a10c-742f6e95764d","Type":"ContainerStarted","Data":"32db76b241b194d741de449ed291f3d27b05a25ade1f880158afb2c9f0830134"} Jul 17 18:10:40.809120 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:40.809074 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" podStartSLOduration=1.809058613 podStartE2EDuration="1.809058613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:40.807505982 +0000 UTC m=+2761.542930061" watchObservedRunningTime="2026-07-17 18:10:40.809058613 +0000 UTC m=+2761.544482661" Jul 17 18:10:41.796738 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:41.796704 2573 generic.go:356] "Generic (PLEG): container finished" podID="9383a51b-695d-4616-a10c-742f6e95764d" containerID="6626b491926b2798e3293a68aa6080bfe21f85df01995285e62463f207a7c8a1" exitCode=6 Jul 17 18:10:41.797130 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:41.796785 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" event={"ID":"9383a51b-695d-4616-a10c-742f6e95764d","Type":"ContainerDied","Data":"6626b491926b2798e3293a68aa6080bfe21f85df01995285e62463f207a7c8a1"} Jul 17 18:10:41.797130 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:41.796955 2573 scope.go:117] "RemoveContainer" containerID="6626b491926b2798e3293a68aa6080bfe21f85df01995285e62463f207a7c8a1" Jul 17 18:10:42.804968 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:42.804932 2573 generic.go:356] "Generic (PLEG): container finished" podID="9383a51b-695d-4616-a10c-742f6e95764d" containerID="95297ee6d1743f5f0835bbc519be9ee902d17886731a98ec1408bb531230b3aa" exitCode=6 Jul 17 18:10:42.805449 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:42.805024 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" event={"ID":"9383a51b-695d-4616-a10c-742f6e95764d","Type":"ContainerDied","Data":"95297ee6d1743f5f0835bbc519be9ee902d17886731a98ec1408bb531230b3aa"} Jul 17 18:10:42.805449 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:42.805097 2573 scope.go:117] "RemoveContainer" containerID="6626b491926b2798e3293a68aa6080bfe21f85df01995285e62463f207a7c8a1" Jul 17 18:10:42.805449 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:42.805322 2573 scope.go:117] "RemoveContainer" containerID="95297ee6d1743f5f0835bbc519be9ee902d17886731a98ec1408bb531230b3aa" Jul 17 18:10:42.805611 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:10:42.805587 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc_odh-ai-gateway-infra(9383a51b-695d-4616-a10c-742f6e95764d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" podUID="9383a51b-695d-4616-a10c-742f6e95764d" Jul 17 18:10:43.811595 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:43.811566 2573 scope.go:117] "RemoveContainer" containerID="95297ee6d1743f5f0835bbc519be9ee902d17886731a98ec1408bb531230b3aa" Jul 17 18:10:43.812060 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:10:43.811847 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc_odh-ai-gateway-infra(9383a51b-695d-4616-a10c-742f6e95764d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" podUID="9383a51b-695d-4616-a10c-742f6e95764d" Jul 17 18:10:51.315556 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:51.315521 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:52.771529 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:52.771494 2573 scope.go:117] "RemoveContainer" containerID="00a65ea7c751357c2783edcb96c07d0b413b72a551842f65eb2e568d995118bd" Jul 17 18:10:53.870174 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:53.870137 2573 generic.go:356] "Generic (PLEG): container finished" podID="f6a911bd-67de-4890-8680-34f41499314c" containerID="1c3f1f95e41838a815111a969dd04c93fbe1ab931b4726f29b489607c2348e6b" exitCode=6 Jul 17 18:10:53.870660 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:53.870206 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt" event={"ID":"f6a911bd-67de-4890-8680-34f41499314c","Type":"ContainerDied","Data":"1c3f1f95e41838a815111a969dd04c93fbe1ab931b4726f29b489607c2348e6b"} Jul 17 18:10:53.870660 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:53.870291 2573 scope.go:117] "RemoveContainer" containerID="00a65ea7c751357c2783edcb96c07d0b413b72a551842f65eb2e568d995118bd" Jul 17 18:10:53.870784 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:53.870698 2573 scope.go:117] "RemoveContainer" containerID="1c3f1f95e41838a815111a969dd04c93fbe1ab931b4726f29b489607c2348e6b" Jul 17 18:10:53.871008 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:10:53.870971 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt_odh-ai-gateway-infra(f6a911bd-67de-4890-8680-34f41499314c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt" podUID="f6a911bd-67de-4890-8680-34f41499314c" Jul 17 18:10:54.771748 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:54.771710 2573 scope.go:117] "RemoveContainer" containerID="95297ee6d1743f5f0835bbc519be9ee902d17886731a98ec1408bb531230b3aa" Jul 17 18:10:54.897970 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:54.897938 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt"] Jul 17 18:10:55.036780 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:55.036720 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt" Jul 17 18:10:55.135708 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:55.135677 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a911bd-67de-4890-8680-34f41499314c-maas-api-service-ca\") pod \"f6a911bd-67de-4890-8680-34f41499314c\" (UID: \"f6a911bd-67de-4890-8680-34f41499314c\") " Jul 17 18:10:55.136011 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:55.135990 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a911bd-67de-4890-8680-34f41499314c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f6a911bd-67de-4890-8680-34f41499314c" (UID: "f6a911bd-67de-4890-8680-34f41499314c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:10:55.236869 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:55.236835 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a911bd-67de-4890-8680-34f41499314c-maas-api-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:10:55.883776 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:55.883742 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt" Jul 17 18:10:55.883976 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:55.883774 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt" event={"ID":"f6a911bd-67de-4890-8680-34f41499314c","Type":"ContainerDied","Data":"15b3712647e87b8ad1116af6aef62931a6ca219f26ae52be5c3ee4cd9b51dc32"} Jul 17 18:10:55.883976 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:55.883835 2573 scope.go:117] "RemoveContainer" containerID="1c3f1f95e41838a815111a969dd04c93fbe1ab931b4726f29b489607c2348e6b" Jul 17 18:10:55.885790 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:55.885762 2573 generic.go:356] "Generic (PLEG): container finished" podID="9383a51b-695d-4616-a10c-742f6e95764d" containerID="130c0824262061d4534255c09a0a9ac25be10e5983b89f542d57b5f4e97724df" exitCode=6 Jul 17 18:10:55.885912 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:55.885831 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" event={"ID":"9383a51b-695d-4616-a10c-742f6e95764d","Type":"ContainerDied","Data":"130c0824262061d4534255c09a0a9ac25be10e5983b89f542d57b5f4e97724df"} Jul 17 18:10:55.886215 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:55.886183 2573 scope.go:117] "RemoveContainer" containerID="130c0824262061d4534255c09a0a9ac25be10e5983b89f542d57b5f4e97724df" Jul 17 18:10:55.886443 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:10:55.886423 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc_odh-ai-gateway-infra(9383a51b-695d-4616-a10c-742f6e95764d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" podUID="9383a51b-695d-4616-a10c-742f6e95764d" Jul 17 18:10:55.892320 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:55.892302 2573 scope.go:117] "RemoveContainer" containerID="95297ee6d1743f5f0835bbc519be9ee902d17886731a98ec1408bb531230b3aa" Jul 17 18:10:55.918071 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:55.918047 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt"] Jul 17 18:10:55.924410 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:55.924388 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49d4a9a-x8rzt"] Jul 17 18:10:56.916791 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:56.916759 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc"] Jul 17 18:10:57.052507 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:57.052481 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" Jul 17 18:10:57.153716 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:57.153687 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9383a51b-695d-4616-a10c-742f6e95764d-maas-api-service-ca\") pod \"9383a51b-695d-4616-a10c-742f6e95764d\" (UID: \"9383a51b-695d-4616-a10c-742f6e95764d\") " Jul 17 18:10:57.154036 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:57.154015 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9383a51b-695d-4616-a10c-742f6e95764d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9383a51b-695d-4616-a10c-742f6e95764d" (UID: "9383a51b-695d-4616-a10c-742f6e95764d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:10:57.254751 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:57.254712 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9383a51b-695d-4616-a10c-742f6e95764d-maas-api-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:10:57.353218 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:57.353185 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:57.454432 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:57.454391 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:57.727046 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:57.726964 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:57.777552 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:57.777516 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6a911bd-67de-4890-8680-34f41499314c" path="/var/lib/kubelet/pods/f6a911bd-67de-4890-8680-34f41499314c/volumes" Jul 17 18:10:57.906921 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:57.906891 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" Jul 17 18:10:57.907109 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:57.906892 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc" event={"ID":"9383a51b-695d-4616-a10c-742f6e95764d","Type":"ContainerDied","Data":"32db76b241b194d741de449ed291f3d27b05a25ade1f880158afb2c9f0830134"} Jul 17 18:10:57.907109 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:57.907044 2573 scope.go:117] "RemoveContainer" containerID="130c0824262061d4534255c09a0a9ac25be10e5983b89f542d57b5f4e97724df" Jul 17 18:10:57.931899 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:57.931870 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc"] Jul 17 18:10:57.942891 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:57.942864 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce07281e-fnkwc"] Jul 17 18:10:58.028672 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.028636 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9"] Jul 17 18:10:58.029049 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.029037 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9383a51b-695d-4616-a10c-742f6e95764d" containerName="revoke-keys" Jul 17 18:10:58.029104 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.029051 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9383a51b-695d-4616-a10c-742f6e95764d" containerName="revoke-keys" Jul 17 18:10:58.029104 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.029069 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6a911bd-67de-4890-8680-34f41499314c" containerName="revoke-keys" Jul 17 18:10:58.029104 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.029074 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6a911bd-67de-4890-8680-34f41499314c" containerName="revoke-keys" Jul 17 18:10:58.029104 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.029081 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6a911bd-67de-4890-8680-34f41499314c" containerName="revoke-keys" Jul 17 18:10:58.029104 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.029087 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6a911bd-67de-4890-8680-34f41499314c" containerName="revoke-keys" Jul 17 18:10:58.029104 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.029097 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9383a51b-695d-4616-a10c-742f6e95764d" containerName="revoke-keys" Jul 17 18:10:58.029104 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.029102 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9383a51b-695d-4616-a10c-742f6e95764d" containerName="revoke-keys" Jul 17 18:10:58.029340 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.029159 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6a911bd-67de-4890-8680-34f41499314c" containerName="revoke-keys" Jul 17 18:10:58.029340 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.029169 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9383a51b-695d-4616-a10c-742f6e95764d" containerName="revoke-keys" Jul 17 18:10:58.029340 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.029176 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9383a51b-695d-4616-a10c-742f6e95764d" containerName="revoke-keys" Jul 17 18:10:58.029340 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.029182 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9383a51b-695d-4616-a10c-742f6e95764d" containerName="revoke-keys" Jul 17 18:10:58.029340 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.029189 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6a911bd-67de-4890-8680-34f41499314c" containerName="revoke-keys" Jul 17 18:10:58.033565 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.033548 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.036997 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.036979 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-54bdd1-openshift-default-dockercfg-lmm9j\"" Jul 17 18:10:58.049787 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.049767 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9"] Jul 17 18:10:58.058186 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.058163 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:58.164859 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.164823 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-podinfo\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.165041 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.164873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337abab0-7def-4cb3-b2ef-baf8e9350b26-istiod-ca-cert\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.165041 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.164905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-workload-socket\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.165041 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.164980 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-credential-socket\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.165041 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.165016 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-workload-certs\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.165174 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.165072 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjws7\" (UniqueName: \"kubernetes.io/projected/337abab0-7def-4cb3-b2ef-baf8e9350b26-kube-api-access-zjws7\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.165174 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.165123 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-token\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.165174 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.165151 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-envoy\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.165274 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.165198 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-data\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.266591 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.266545 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-token\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.266591 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.266595 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-envoy\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.266882 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.266664 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-data\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.266882 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.266708 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-podinfo\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.266882 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.266748 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337abab0-7def-4cb3-b2ef-baf8e9350b26-istiod-ca-cert\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.266882 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.266774 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-workload-socket\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.266882 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.266815 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-credential-socket\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.266882 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.266843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-workload-certs\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.267181 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.266995 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjws7\" (UniqueName: \"kubernetes.io/projected/337abab0-7def-4cb3-b2ef-baf8e9350b26-kube-api-access-zjws7\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.267181 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.267136 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-data\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.267288 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.267184 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-credential-socket\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.267288 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.267214 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-workload-certs\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.267400 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.267282 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-workload-socket\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.267530 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.267511 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337abab0-7def-4cb3-b2ef-baf8e9350b26-istiod-ca-cert\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.269109 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.269085 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-envoy\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.269275 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.269256 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-podinfo\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.276613 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.276592 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-token\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.276794 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.276774 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjws7\" (UniqueName: \"kubernetes.io/projected/337abab0-7def-4cb3-b2ef-baf8e9350b26-kube-api-access-zjws7\") pod \"e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.344406 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.344319 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:58.489122 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.489082 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9"] Jul 17 18:10:58.493693 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:10:58.493665 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337abab0_7def_4cb3_b2ef_baf8e9350b26.slice/crio-f7d9ad227e5ef5956e225d2089d54eed0d9e831aa5312009e4c930891257a266 WatchSource:0}: Error finding container f7d9ad227e5ef5956e225d2089d54eed0d9e831aa5312009e4c930891257a266: Status 404 returned error can't find the container with id f7d9ad227e5ef5956e225d2089d54eed0d9e831aa5312009e4c930891257a266 Jul 17 18:10:58.496057 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.496023 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:10:58.496161 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.496103 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:10:58.496161 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.496152 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:10:58.498684 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.498658 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:58.914206 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.914168 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" event={"ID":"337abab0-7def-4cb3-b2ef-baf8e9350b26","Type":"ContainerStarted","Data":"527879b46f5455bab877405b9162b7e8264978fc40fae5e7c6a9023701d4ed10"} Jul 17 18:10:58.914425 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.914217 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" event={"ID":"337abab0-7def-4cb3-b2ef-baf8e9350b26","Type":"ContainerStarted","Data":"f7d9ad227e5ef5956e225d2089d54eed0d9e831aa5312009e4c930891257a266"} Jul 17 18:10:58.937370 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:58.937309 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" podStartSLOduration=0.937295525 podStartE2EDuration="937.295525ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:58.936019559 +0000 UTC m=+2779.671443643" watchObservedRunningTime="2026-07-17 18:10:58.937295525 +0000 UTC m=+2779.672719574" Jul 17 18:10:59.345051 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:59.345013 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:59.349965 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:59.349939 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:59.777726 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:59.777693 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9383a51b-695d-4616-a10c-742f6e95764d" path="/var/lib/kubelet/pods/9383a51b-695d-4616-a10c-742f6e95764d/volumes" Jul 17 18:10:59.919140 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:59.919106 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:10:59.920011 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:10:59.919985 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:11:00.953030 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:00.952941 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:01.042186 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.042150 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt"] Jul 17 18:11:01.042819 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.042799 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6a911bd-67de-4890-8680-34f41499314c" containerName="revoke-keys" Jul 17 18:11:01.042868 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.042824 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6a911bd-67de-4890-8680-34f41499314c" containerName="revoke-keys" Jul 17 18:11:01.042868 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.042849 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9383a51b-695d-4616-a10c-742f6e95764d" containerName="revoke-keys" Jul 17 18:11:01.042868 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.042858 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9383a51b-695d-4616-a10c-742f6e95764d" containerName="revoke-keys" Jul 17 18:11:01.042996 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.042983 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6a911bd-67de-4890-8680-34f41499314c" containerName="revoke-keys" Jul 17 18:11:01.046788 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.046767 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" Jul 17 18:11:01.051972 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.051951 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-54bdd1-serving-cert\"" Jul 17 18:11:01.052715 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.052687 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:01.059250 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.059209 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt"] Jul 17 18:11:01.148914 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.148879 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:01.195408 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.195378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e8656e8-5c3b-4809-8366-99ead858ed26-maas-api-tls\") pod \"maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt\" (UID: \"2e8656e8-5c3b-4809-8366-99ead858ed26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" Jul 17 18:11:01.195602 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.195415 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8656e8-5c3b-4809-8366-99ead858ed26-openshift-service-ca\") pod \"maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt\" (UID: \"2e8656e8-5c3b-4809-8366-99ead858ed26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" Jul 17 18:11:01.195602 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.195463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcfwp\" (UniqueName: \"kubernetes.io/projected/2e8656e8-5c3b-4809-8366-99ead858ed26-kube-api-access-vcfwp\") pod \"maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt\" (UID: \"2e8656e8-5c3b-4809-8366-99ead858ed26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" Jul 17 18:11:01.249845 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.249813 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:01.296104 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.296064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e8656e8-5c3b-4809-8366-99ead858ed26-maas-api-tls\") pod \"maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt\" (UID: \"2e8656e8-5c3b-4809-8366-99ead858ed26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" Jul 17 18:11:01.296104 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.296106 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8656e8-5c3b-4809-8366-99ead858ed26-openshift-service-ca\") pod \"maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt\" (UID: \"2e8656e8-5c3b-4809-8366-99ead858ed26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" Jul 17 18:11:01.296350 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.296172 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcfwp\" (UniqueName: \"kubernetes.io/projected/2e8656e8-5c3b-4809-8366-99ead858ed26-kube-api-access-vcfwp\") pod \"maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt\" (UID: \"2e8656e8-5c3b-4809-8366-99ead858ed26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" Jul 17 18:11:01.296841 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.296820 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8656e8-5c3b-4809-8366-99ead858ed26-openshift-service-ca\") pod \"maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt\" (UID: \"2e8656e8-5c3b-4809-8366-99ead858ed26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" Jul 17 18:11:01.298452 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.298433 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e8656e8-5c3b-4809-8366-99ead858ed26-maas-api-tls\") pod \"maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt\" (UID: \"2e8656e8-5c3b-4809-8366-99ead858ed26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" Jul 17 18:11:01.305317 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.305288 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcfwp\" (UniqueName: \"kubernetes.io/projected/2e8656e8-5c3b-4809-8366-99ead858ed26-kube-api-access-vcfwp\") pod \"maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt\" (UID: \"2e8656e8-5c3b-4809-8366-99ead858ed26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" Jul 17 18:11:01.360313 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.360274 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" Jul 17 18:11:01.498958 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.498934 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt"] Jul 17 18:11:01.500924 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:11:01.500857 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8656e8_5c3b_4809_8366_99ead858ed26.slice/crio-45fbecce1239dcb6adeaed21981d705dd03a74ca8097bcd1ee9c09cc136f0f17 WatchSource:0}: Error finding container 45fbecce1239dcb6adeaed21981d705dd03a74ca8097bcd1ee9c09cc136f0f17: Status 404 returned error can't find the container with id 45fbecce1239dcb6adeaed21981d705dd03a74ca8097bcd1ee9c09cc136f0f17 Jul 17 18:11:01.712041 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.712006 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:01.813513 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.813436 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:01.931529 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:01.931502 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" event={"ID":"2e8656e8-5c3b-4809-8366-99ead858ed26","Type":"ContainerStarted","Data":"45fbecce1239dcb6adeaed21981d705dd03a74ca8097bcd1ee9c09cc136f0f17"} Jul 17 18:11:02.939538 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:02.939506 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" event={"ID":"2e8656e8-5c3b-4809-8366-99ead858ed26","Type":"ContainerStarted","Data":"8239b7083a327fa4880131f1508ca9b3653428c23982705424769f3950ac38dc"} Jul 17 18:11:02.939952 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:02.939719 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" Jul 17 18:11:02.960532 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:02.960484 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" podStartSLOduration=1.554859806 podStartE2EDuration="1.960467452s" totalImagesPullingTime="405.607646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:02.958495927 +0000 UTC m=+2783.693920002" watchObservedRunningTime="2026-07-17 18:11:02.960467452 +0000 UTC m=+2783.695891501" Jul 17 18:11:05.336921 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:05.336863 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:05.667818 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:05.667722 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:06.045383 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.045347 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2"] Jul 17 18:11:06.048886 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.048866 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.052651 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.052615 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c50cdb-openshift-default-dockercfg-jfp7c\"" Jul 17 18:11:06.070644 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.070595 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2"] Jul 17 18:11:06.079152 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.079121 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:06.141854 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.141816 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-envoy\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.142033 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.141869 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-token\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.142033 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.141918 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-data\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.142033 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.141972 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-workload-socket\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.142033 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.142001 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-workload-certs\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.142159 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.142055 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9rcg\" (UniqueName: \"kubernetes.io/projected/b97187d2-c474-4e5d-9418-b32d4a37a148-kube-api-access-r9rcg\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.142159 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.142103 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97187d2-c474-4e5d-9418-b32d4a37a148-istiod-ca-cert\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.142159 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.142140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-credential-socket\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.142251 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.142172 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-podinfo\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.243643 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.243580 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-token\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.243643 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.243616 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-data\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.243881 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.243692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-workload-socket\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.243881 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.243721 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-workload-certs\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.243881 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.243751 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9rcg\" (UniqueName: \"kubernetes.io/projected/b97187d2-c474-4e5d-9418-b32d4a37a148-kube-api-access-r9rcg\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.243881 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.243801 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97187d2-c474-4e5d-9418-b32d4a37a148-istiod-ca-cert\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.243881 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.243849 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-credential-socket\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.243881 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.243871 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-podinfo\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.244167 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.243911 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-envoy\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.244167 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.244132 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-data\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.244279 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.244213 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-workload-certs\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.244336 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.244307 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-credential-socket\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.244438 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.244423 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-workload-socket\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.245082 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.245054 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97187d2-c474-4e5d-9418-b32d4a37a148-istiod-ca-cert\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.246264 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.246244 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-envoy\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.246387 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.246370 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-podinfo\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.253059 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.253040 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-token\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.253276 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.253254 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9rcg\" (UniqueName: \"kubernetes.io/projected/b97187d2-c474-4e5d-9418-b32d4a37a148-kube-api-access-r9rcg\") pod \"e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.360256 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.360169 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:06.484591 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.484556 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:06.505073 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.505050 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2"] Jul 17 18:11:06.510493 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:11:06.507607 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97187d2_c474_4e5d_9418_b32d4a37a148.slice/crio-ff2f8c74b5940e9775eecbdcd53969907697e24e1218a35122555e8a921011bd WatchSource:0}: Error finding container ff2f8c74b5940e9775eecbdcd53969907697e24e1218a35122555e8a921011bd: Status 404 returned error can't find the container with id ff2f8c74b5940e9775eecbdcd53969907697e24e1218a35122555e8a921011bd Jul 17 18:11:06.512292 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.512254 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:11:06.512381 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.512338 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:11:06.512438 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.512383 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:11:06.961456 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.961414 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" event={"ID":"b97187d2-c474-4e5d-9418-b32d4a37a148","Type":"ContainerStarted","Data":"321729487f770e3fceb6d369f0f9ad390cb3e6886ca72935699089dd06807d45"} Jul 17 18:11:06.961779 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.961471 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" event={"ID":"b97187d2-c474-4e5d-9418-b32d4a37a148","Type":"ContainerStarted","Data":"ff2f8c74b5940e9775eecbdcd53969907697e24e1218a35122555e8a921011bd"} Jul 17 18:11:06.982608 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:06.982551 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" podStartSLOduration=0.982533762 podStartE2EDuration="982.533762ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:06.982442416 +0000 UTC m=+2787.717866465" watchObservedRunningTime="2026-07-17 18:11:06.982533762 +0000 UTC m=+2787.717957811" Jul 17 18:11:07.360632 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:07.360591 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:07.362132 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:07.362104 2573 patch_prober.go:28] interesting pod/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 17 18:11:07.362264 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:07.362153 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" podUID="b97187d2-c474-4e5d-9418-b32d4a37a148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 17 18:11:08.364958 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:08.364927 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:08.941749 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:08.941714 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:08.949512 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:08.949488 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" Jul 17 18:11:08.972513 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:08.972484 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:08.973475 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:08.973455 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:09.037811 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.037773 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:09.275546 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.275514 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5"] Jul 17 18:11:09.279426 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.279404 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" Jul 17 18:11:09.282747 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.282729 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c50cdb-serving-cert\"" Jul 17 18:11:09.296343 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.296320 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5"] Jul 17 18:11:09.323252 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.323204 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:09.374528 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.374494 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7q86\" (UniqueName: \"kubernetes.io/projected/af279baa-9613-456a-8870-6fd9f306eaaa-kube-api-access-f7q86\") pod \"maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5\" (UID: \"af279baa-9613-456a-8870-6fd9f306eaaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" Jul 17 18:11:09.374953 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.374553 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af279baa-9613-456a-8870-6fd9f306eaaa-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5\" (UID: \"af279baa-9613-456a-8870-6fd9f306eaaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" Jul 17 18:11:09.374953 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.374695 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af279baa-9613-456a-8870-6fd9f306eaaa-maas-api-tls\") pod \"maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5\" (UID: \"af279baa-9613-456a-8870-6fd9f306eaaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" Jul 17 18:11:09.476013 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.475971 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af279baa-9613-456a-8870-6fd9f306eaaa-maas-api-tls\") pod \"maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5\" (UID: \"af279baa-9613-456a-8870-6fd9f306eaaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" Jul 17 18:11:09.476205 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.476064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7q86\" (UniqueName: \"kubernetes.io/projected/af279baa-9613-456a-8870-6fd9f306eaaa-kube-api-access-f7q86\") pod \"maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5\" (UID: \"af279baa-9613-456a-8870-6fd9f306eaaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" Jul 17 18:11:09.476205 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.476113 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af279baa-9613-456a-8870-6fd9f306eaaa-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5\" (UID: \"af279baa-9613-456a-8870-6fd9f306eaaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" Jul 17 18:11:09.476205 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:11:09.476126 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-serving-cert: secret "maas-api-e2e-inf-b-c50cdb-serving-cert" not found Jul 17 18:11:09.476205 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:11:09.476187 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af279baa-9613-456a-8870-6fd9f306eaaa-maas-api-tls podName:af279baa-9613-456a-8870-6fd9f306eaaa nodeName:}" failed. No retries permitted until 2026-07-17 18:11:09.976171295 +0000 UTC m=+2790.711595323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af279baa-9613-456a-8870-6fd9f306eaaa-maas-api-tls") pod "maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" (UID: "af279baa-9613-456a-8870-6fd9f306eaaa") : secret "maas-api-e2e-inf-b-c50cdb-serving-cert" not found Jul 17 18:11:09.476764 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.476745 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af279baa-9613-456a-8870-6fd9f306eaaa-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5\" (UID: \"af279baa-9613-456a-8870-6fd9f306eaaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" Jul 17 18:11:09.490518 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.490495 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7q86\" (UniqueName: \"kubernetes.io/projected/af279baa-9613-456a-8870-6fd9f306eaaa-kube-api-access-f7q86\") pod \"maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5\" (UID: \"af279baa-9613-456a-8870-6fd9f306eaaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" Jul 17 18:11:09.560582 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.560492 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:09.894554 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.894471 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:09.980898 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.980861 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af279baa-9613-456a-8870-6fd9f306eaaa-maas-api-tls\") pod \"maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5\" (UID: \"af279baa-9613-456a-8870-6fd9f306eaaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" Jul 17 18:11:09.983259 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:09.983230 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af279baa-9613-456a-8870-6fd9f306eaaa-maas-api-tls\") pod \"maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5\" (UID: \"af279baa-9613-456a-8870-6fd9f306eaaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" Jul 17 18:11:10.025304 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:10.025271 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:10.190497 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:10.190404 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" Jul 17 18:11:10.343821 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:10.343793 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5"] Jul 17 18:11:10.345812 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:11:10.345787 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf279baa_9613_456a_8870_6fd9f306eaaa.slice/crio-f5ecbf5df48070f93be89122a08972202c3d01c95106c091cf77a5d28d4130cd WatchSource:0}: Error finding container f5ecbf5df48070f93be89122a08972202c3d01c95106c091cf77a5d28d4130cd: Status 404 returned error can't find the container with id f5ecbf5df48070f93be89122a08972202c3d01c95106c091cf77a5d28d4130cd Jul 17 18:11:10.983898 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:10.983856 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" event={"ID":"af279baa-9613-456a-8870-6fd9f306eaaa","Type":"ContainerStarted","Data":"2b248289f4a0821c5c621dfbd4252b4e3a485a6ffdce594dc61ffae79ebf3a90"} Jul 17 18:11:10.984387 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:10.984090 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" Jul 17 18:11:10.984387 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:10.984130 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" event={"ID":"af279baa-9613-456a-8870-6fd9f306eaaa","Type":"ContainerStarted","Data":"f5ecbf5df48070f93be89122a08972202c3d01c95106c091cf77a5d28d4130cd"} Jul 17 18:11:11.006012 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:11.005943 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" podStartSLOduration=1.581203329 podStartE2EDuration="2.005897334s" totalImagesPullingTime="424.694005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:11.001218703 +0000 UTC m=+2791.736642781" watchObservedRunningTime="2026-07-17 18:11:11.005897334 +0000 UTC m=+2791.741321385" Jul 17 18:11:13.414425 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:13.414375 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:15.960826 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:15.960784 2573 patch_prober.go:28] interesting pod/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 17 18:11:15.961207 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:15.960861 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" podUID="337abab0-7def-4cb3-b2ef-baf8e9350b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 17 18:11:16.994636 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:16.994589 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" Jul 17 18:11:18.030713 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:18.030677 2573 generic.go:356] "Generic (PLEG): container finished" podID="337abab0-7def-4cb3-b2ef-baf8e9350b26" containerID="527879b46f5455bab877405b9162b7e8264978fc40fae5e7c6a9023701d4ed10" exitCode=0 Jul 17 18:11:18.031134 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:18.030753 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" event={"ID":"337abab0-7def-4cb3-b2ef-baf8e9350b26","Type":"ContainerDied","Data":"527879b46f5455bab877405b9162b7e8264978fc40fae5e7c6a9023701d4ed10"} Jul 17 18:11:18.031176 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:18.031160 2573 scope.go:117] "RemoveContainer" containerID="527879b46f5455bab877405b9162b7e8264978fc40fae5e7c6a9023701d4ed10" Jul 17 18:11:18.032162 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:18.032136 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:11:18.032215 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:18.032202 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:11:18.032250 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:18.032237 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:11:19.042400 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:19.042363 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" event={"ID":"337abab0-7def-4cb3-b2ef-baf8e9350b26","Type":"ContainerStarted","Data":"21b6cd5be57dab5c9ddf852ff032085ea790f2ec987806174d4b7dcf728bcd0b"} Jul 17 18:11:19.345199 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:19.345024 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:11:19.345875 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:19.345848 2573 patch_prober.go:28] interesting pod/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 17 18:11:19.346114 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:19.346082 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" podUID="337abab0-7def-4cb3-b2ef-baf8e9350b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 17 18:11:20.034328 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:20.034258 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:20.345143 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:20.345035 2573 patch_prober.go:28] interesting pod/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 17 18:11:20.345143 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:20.345126 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" podUID="337abab0-7def-4cb3-b2ef-baf8e9350b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 17 18:11:21.345465 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:21.345427 2573 patch_prober.go:28] interesting pod/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 17 18:11:21.345883 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:21.345480 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" podUID="337abab0-7def-4cb3-b2ef-baf8e9350b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 17 18:11:22.345653 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:22.345592 2573 patch_prober.go:28] interesting pod/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 17 18:11:22.346094 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:22.345684 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" podUID="337abab0-7def-4cb3-b2ef-baf8e9350b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 17 18:11:23.349827 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:23.349791 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:11:23.350289 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:23.350162 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:11:23.351065 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:23.351042 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:11:23.592712 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:23.592671 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:23.691349 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:23.691264 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:24.235151 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:24.234460 2573 patch_prober.go:28] interesting pod/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": read tcp 10.134.0.2:35410->10.134.0.60:15021: read: connection reset by peer" start-of-body= Jul 17 18:11:24.235151 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:24.234525 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" podUID="b97187d2-c474-4e5d-9418-b32d4a37a148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": read tcp 10.134.0.2:35410->10.134.0.60:15021: read: connection reset by peer" Jul 17 18:11:25.086515 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:25.086478 2573 generic.go:356] "Generic (PLEG): container finished" podID="b97187d2-c474-4e5d-9418-b32d4a37a148" containerID="321729487f770e3fceb6d369f0f9ad390cb3e6886ca72935699089dd06807d45" exitCode=0 Jul 17 18:11:25.087031 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:25.086551 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" event={"ID":"b97187d2-c474-4e5d-9418-b32d4a37a148","Type":"ContainerDied","Data":"321729487f770e3fceb6d369f0f9ad390cb3e6886ca72935699089dd06807d45"} Jul 17 18:11:25.087120 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:25.087103 2573 scope.go:117] "RemoveContainer" containerID="321729487f770e3fceb6d369f0f9ad390cb3e6886ca72935699089dd06807d45" Jul 17 18:11:25.088111 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:25.088084 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:11:25.088197 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:25.088151 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:11:25.088197 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:25.088186 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:11:26.093753 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:26.093717 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" event={"ID":"b97187d2-c474-4e5d-9418-b32d4a37a148","Type":"ContainerStarted","Data":"3ccf7087f8ec02bd5dd650a618582d205145c780866edcf31db75af6040f9fdd"} Jul 17 18:11:26.361417 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:26.361321 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:26.361580 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:26.361541 2573 patch_prober.go:28] interesting pod/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 17 18:11:26.361660 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:26.361587 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" podUID="b97187d2-c474-4e5d-9418-b32d4a37a148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 17 18:11:27.361214 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:27.361177 2573 patch_prober.go:28] interesting pod/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 17 18:11:27.361650 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:27.361244 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" podUID="b97187d2-c474-4e5d-9418-b32d4a37a148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 17 18:11:28.361093 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:28.361055 2573 patch_prober.go:28] interesting pod/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 17 18:11:28.361266 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:28.361115 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" podUID="b97187d2-c474-4e5d-9418-b32d4a37a148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 17 18:11:29.360541 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:29.360501 2573 patch_prober.go:28] interesting pod/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 17 18:11:29.360755 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:29.360558 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" podUID="b97187d2-c474-4e5d-9418-b32d4a37a148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 17 18:11:30.364642 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:30.364597 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:30.365178 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:30.365156 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:30.365787 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:30.365768 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:11:30.899986 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:30.899946 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:31.001996 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:31.001956 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:32.837867 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:32.837831 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:47.629085 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:11:47.629042 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:06.753318 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:06.753257 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:07.674976 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:07.674942 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:08.307093 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:08.307050 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:08.406425 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:08.406391 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:18.206721 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:18.206685 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt"] Jul 17 18:12:18.207250 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:18.206962 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" podUID="2e8656e8-5c3b-4809-8366-99ead858ed26" containerName="maas-api" containerID="cri-o://8239b7083a327fa4880131f1508ca9b3653428c23982705424769f3950ac38dc" gracePeriod=30 Jul 17 18:12:18.233927 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:18.233900 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:18.235718 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:12:18.235696 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8656e8_5c3b_4809_8366_99ead858ed26.slice/crio-8239b7083a327fa4880131f1508ca9b3653428c23982705424769f3950ac38dc.scope\": RecentStats: unable to find data in memory cache]" Jul 17 18:12:18.375187 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:18.375159 2573 generic.go:356] "Generic (PLEG): container finished" podID="2e8656e8-5c3b-4809-8366-99ead858ed26" containerID="8239b7083a327fa4880131f1508ca9b3653428c23982705424769f3950ac38dc" exitCode=0 Jul 17 18:12:18.375344 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:18.375237 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" event={"ID":"2e8656e8-5c3b-4809-8366-99ead858ed26","Type":"ContainerDied","Data":"8239b7083a327fa4880131f1508ca9b3653428c23982705424769f3950ac38dc"} Jul 17 18:12:18.457606 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:18.457549 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" Jul 17 18:12:18.518671 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:18.518638 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcfwp\" (UniqueName: \"kubernetes.io/projected/2e8656e8-5c3b-4809-8366-99ead858ed26-kube-api-access-vcfwp\") pod \"2e8656e8-5c3b-4809-8366-99ead858ed26\" (UID: \"2e8656e8-5c3b-4809-8366-99ead858ed26\") " Jul 17 18:12:18.518817 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:18.518726 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e8656e8-5c3b-4809-8366-99ead858ed26-maas-api-tls\") pod \"2e8656e8-5c3b-4809-8366-99ead858ed26\" (UID: \"2e8656e8-5c3b-4809-8366-99ead858ed26\") " Jul 17 18:12:18.518817 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:18.518792 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8656e8-5c3b-4809-8366-99ead858ed26-openshift-service-ca\") pod \"2e8656e8-5c3b-4809-8366-99ead858ed26\" (UID: \"2e8656e8-5c3b-4809-8366-99ead858ed26\") " Jul 17 18:12:18.519177 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:18.519151 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e8656e8-5c3b-4809-8366-99ead858ed26-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e8656e8-5c3b-4809-8366-99ead858ed26" (UID: "2e8656e8-5c3b-4809-8366-99ead858ed26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:12:18.520657 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:18.520612 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8656e8-5c3b-4809-8366-99ead858ed26-kube-api-access-vcfwp" (OuterVolumeSpecName: "kube-api-access-vcfwp") pod "2e8656e8-5c3b-4809-8366-99ead858ed26" (UID: "2e8656e8-5c3b-4809-8366-99ead858ed26"). InnerVolumeSpecName "kube-api-access-vcfwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:12:18.520826 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:18.520802 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e8656e8-5c3b-4809-8366-99ead858ed26-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e8656e8-5c3b-4809-8366-99ead858ed26" (UID: "2e8656e8-5c3b-4809-8366-99ead858ed26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:12:18.620014 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:18.619981 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcfwp\" (UniqueName: \"kubernetes.io/projected/2e8656e8-5c3b-4809-8366-99ead858ed26-kube-api-access-vcfwp\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:12:18.620014 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:18.620009 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e8656e8-5c3b-4809-8366-99ead858ed26-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:12:18.620014 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:18.620019 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8656e8-5c3b-4809-8366-99ead858ed26-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:12:19.380976 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:19.380947 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" Jul 17 18:12:19.380976 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:19.380958 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt" event={"ID":"2e8656e8-5c3b-4809-8366-99ead858ed26","Type":"ContainerDied","Data":"45fbecce1239dcb6adeaed21981d705dd03a74ca8097bcd1ee9c09cc136f0f17"} Jul 17 18:12:19.381468 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:19.381019 2573 scope.go:117] "RemoveContainer" containerID="8239b7083a327fa4880131f1508ca9b3653428c23982705424769f3950ac38dc" Jul 17 18:12:19.408648 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:19.408604 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt"] Jul 17 18:12:19.414367 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:19.414343 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-54bdd1-d8cb87854-2mljt"] Jul 17 18:12:19.778227 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:19.778192 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e8656e8-5c3b-4809-8366-99ead858ed26" path="/var/lib/kubelet/pods/2e8656e8-5c3b-4809-8366-99ead858ed26/volumes" Jul 17 18:12:22.874294 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:22.874263 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:27.939613 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:27.939578 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:28.039369 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:28.039332 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:29.025305 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:29.025252 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:29.126810 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:29.126778 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:33.596031 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:33.595991 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9"] Jul 17 18:12:33.596473 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:33.596215 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" podUID="337abab0-7def-4cb3-b2ef-baf8e9350b26" containerName="istio-proxy" containerID="cri-o://21b6cd5be57dab5c9ddf852ff032085ea790f2ec987806174d4b7dcf728bcd0b" gracePeriod=30 Jul 17 18:12:33.642034 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:33.641997 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:33.804531 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:33.804501 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:34.623979 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:34.623944 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:34.717637 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:34.717588 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:38.352894 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:38.352862 2573 patch_prober.go:28] interesting pod/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 18:12:38.353261 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:38.352917 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" podUID="337abab0-7def-4cb3-b2ef-baf8e9350b26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 18:12:38.857391 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:38.857364 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:12:39.026394 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.026360 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-credential-socket\") pod \"337abab0-7def-4cb3-b2ef-baf8e9350b26\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " Jul 17 18:12:39.026394 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.026394 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-envoy\") pod \"337abab0-7def-4cb3-b2ef-baf8e9350b26\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " Jul 17 18:12:39.026712 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.026444 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337abab0-7def-4cb3-b2ef-baf8e9350b26-istiod-ca-cert\") pod \"337abab0-7def-4cb3-b2ef-baf8e9350b26\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " Jul 17 18:12:39.026712 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.026502 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-workload-certs\") pod \"337abab0-7def-4cb3-b2ef-baf8e9350b26\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " Jul 17 18:12:39.026712 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.026524 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-token\") pod \"337abab0-7def-4cb3-b2ef-baf8e9350b26\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " Jul 17 18:12:39.026712 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.026603 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-data\") pod \"337abab0-7def-4cb3-b2ef-baf8e9350b26\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " Jul 17 18:12:39.026712 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.026679 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-podinfo\") pod \"337abab0-7def-4cb3-b2ef-baf8e9350b26\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " Jul 17 18:12:39.026970 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.026718 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjws7\" (UniqueName: \"kubernetes.io/projected/337abab0-7def-4cb3-b2ef-baf8e9350b26-kube-api-access-zjws7\") pod \"337abab0-7def-4cb3-b2ef-baf8e9350b26\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " Jul 17 18:12:39.026970 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.026760 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-workload-socket\") pod \"337abab0-7def-4cb3-b2ef-baf8e9350b26\" (UID: \"337abab0-7def-4cb3-b2ef-baf8e9350b26\") " Jul 17 18:12:39.026970 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.026754 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "337abab0-7def-4cb3-b2ef-baf8e9350b26" (UID: "337abab0-7def-4cb3-b2ef-baf8e9350b26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:12:39.026970 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.026950 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "337abab0-7def-4cb3-b2ef-baf8e9350b26" (UID: "337abab0-7def-4cb3-b2ef-baf8e9350b26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:12:39.027190 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.027028 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337abab0-7def-4cb3-b2ef-baf8e9350b26-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "337abab0-7def-4cb3-b2ef-baf8e9350b26" (UID: "337abab0-7def-4cb3-b2ef-baf8e9350b26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:12:39.027248 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.027198 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:12:39.027248 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.027217 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337abab0-7def-4cb3-b2ef-baf8e9350b26-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:12:39.027248 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.027232 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:12:39.027248 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.027235 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "337abab0-7def-4cb3-b2ef-baf8e9350b26" (UID: "337abab0-7def-4cb3-b2ef-baf8e9350b26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:12:39.027504 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.027479 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-data" (OuterVolumeSpecName: "istio-data") pod "337abab0-7def-4cb3-b2ef-baf8e9350b26" (UID: "337abab0-7def-4cb3-b2ef-baf8e9350b26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:12:39.028858 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.028831 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337abab0-7def-4cb3-b2ef-baf8e9350b26-kube-api-access-zjws7" (OuterVolumeSpecName: "kube-api-access-zjws7") pod "337abab0-7def-4cb3-b2ef-baf8e9350b26" (UID: "337abab0-7def-4cb3-b2ef-baf8e9350b26"). InnerVolumeSpecName "kube-api-access-zjws7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:12:39.028966 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.028893 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-token" (OuterVolumeSpecName: "istio-token") pod "337abab0-7def-4cb3-b2ef-baf8e9350b26" (UID: "337abab0-7def-4cb3-b2ef-baf8e9350b26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:12:39.029170 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.029153 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "337abab0-7def-4cb3-b2ef-baf8e9350b26" (UID: "337abab0-7def-4cb3-b2ef-baf8e9350b26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:12:39.029227 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.029162 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "337abab0-7def-4cb3-b2ef-baf8e9350b26" (UID: "337abab0-7def-4cb3-b2ef-baf8e9350b26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 18:12:39.127944 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.127911 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:12:39.127944 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.127938 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjws7\" (UniqueName: \"kubernetes.io/projected/337abab0-7def-4cb3-b2ef-baf8e9350b26-kube-api-access-zjws7\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:12:39.127944 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.127947 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:12:39.128170 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.127958 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:12:39.128170 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.127968 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:12:39.128170 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.127977 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337abab0-7def-4cb3-b2ef-baf8e9350b26-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:12:39.484030 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.484000 2573 generic.go:356] "Generic (PLEG): container finished" podID="337abab0-7def-4cb3-b2ef-baf8e9350b26" containerID="21b6cd5be57dab5c9ddf852ff032085ea790f2ec987806174d4b7dcf728bcd0b" exitCode=0 Jul 17 18:12:39.484427 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.484048 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" Jul 17 18:12:39.484427 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.484079 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" event={"ID":"337abab0-7def-4cb3-b2ef-baf8e9350b26","Type":"ContainerDied","Data":"21b6cd5be57dab5c9ddf852ff032085ea790f2ec987806174d4b7dcf728bcd0b"} Jul 17 18:12:39.484427 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.484132 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9" event={"ID":"337abab0-7def-4cb3-b2ef-baf8e9350b26","Type":"ContainerDied","Data":"f7d9ad227e5ef5956e225d2089d54eed0d9e831aa5312009e4c930891257a266"} Jul 17 18:12:39.484427 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.484153 2573 scope.go:117] "RemoveContainer" containerID="21b6cd5be57dab5c9ddf852ff032085ea790f2ec987806174d4b7dcf728bcd0b" Jul 17 18:12:39.501374 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.501352 2573 scope.go:117] "RemoveContainer" containerID="527879b46f5455bab877405b9162b7e8264978fc40fae5e7c6a9023701d4ed10" Jul 17 18:12:39.509547 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.509525 2573 scope.go:117] "RemoveContainer" containerID="21b6cd5be57dab5c9ddf852ff032085ea790f2ec987806174d4b7dcf728bcd0b" Jul 17 18:12:39.509924 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:12:39.509898 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b6cd5be57dab5c9ddf852ff032085ea790f2ec987806174d4b7dcf728bcd0b\": container with ID starting with 21b6cd5be57dab5c9ddf852ff032085ea790f2ec987806174d4b7dcf728bcd0b not found: ID does not exist" containerID="21b6cd5be57dab5c9ddf852ff032085ea790f2ec987806174d4b7dcf728bcd0b" Jul 17 18:12:39.510041 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.509928 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b6cd5be57dab5c9ddf852ff032085ea790f2ec987806174d4b7dcf728bcd0b"} err="failed to get container status \"21b6cd5be57dab5c9ddf852ff032085ea790f2ec987806174d4b7dcf728bcd0b\": rpc error: code = NotFound desc = could not find container \"21b6cd5be57dab5c9ddf852ff032085ea790f2ec987806174d4b7dcf728bcd0b\": container with ID starting with 21b6cd5be57dab5c9ddf852ff032085ea790f2ec987806174d4b7dcf728bcd0b not found: ID does not exist" Jul 17 18:12:39.510041 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.509947 2573 scope.go:117] "RemoveContainer" containerID="527879b46f5455bab877405b9162b7e8264978fc40fae5e7c6a9023701d4ed10" Jul 17 18:12:39.510186 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:12:39.510168 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527879b46f5455bab877405b9162b7e8264978fc40fae5e7c6a9023701d4ed10\": container with ID starting with 527879b46f5455bab877405b9162b7e8264978fc40fae5e7c6a9023701d4ed10 not found: ID does not exist" containerID="527879b46f5455bab877405b9162b7e8264978fc40fae5e7c6a9023701d4ed10" Jul 17 18:12:39.510245 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.510190 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527879b46f5455bab877405b9162b7e8264978fc40fae5e7c6a9023701d4ed10"} err="failed to get container status \"527879b46f5455bab877405b9162b7e8264978fc40fae5e7c6a9023701d4ed10\": rpc error: code = NotFound desc = could not find container \"527879b46f5455bab877405b9162b7e8264978fc40fae5e7c6a9023701d4ed10\": container with ID starting with 527879b46f5455bab877405b9162b7e8264978fc40fae5e7c6a9023701d4ed10 not found: ID does not exist" Jul 17 18:12:39.511061 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.511039 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9"] Jul 17 18:12:39.515296 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.515274 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-54bdd1-openshift-default-78699d8789-8b6x9"] Jul 17 18:12:39.778488 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:39.778413 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="337abab0-7def-4cb3-b2ef-baf8e9350b26" path="/var/lib/kubelet/pods/337abab0-7def-4cb3-b2ef-baf8e9350b26/volumes" Jul 17 18:12:44.393551 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:44.393521 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5"] Jul 17 18:12:44.394009 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:44.393811 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" podUID="af279baa-9613-456a-8870-6fd9f306eaaa" containerName="maas-api" containerID="cri-o://2b248289f4a0821c5c621dfbd4252b4e3a485a6ffdce594dc61ffae79ebf3a90" gracePeriod=30 Jul 17 18:12:44.420240 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:44.420217 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:44.512180 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:44.512155 2573 generic.go:356] "Generic (PLEG): container finished" podID="af279baa-9613-456a-8870-6fd9f306eaaa" containerID="2b248289f4a0821c5c621dfbd4252b4e3a485a6ffdce594dc61ffae79ebf3a90" exitCode=0 Jul 17 18:12:44.512275 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:44.512224 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" event={"ID":"af279baa-9613-456a-8870-6fd9f306eaaa","Type":"ContainerDied","Data":"2b248289f4a0821c5c621dfbd4252b4e3a485a6ffdce594dc61ffae79ebf3a90"} Jul 17 18:12:44.647491 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:44.647425 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" Jul 17 18:12:44.681209 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:44.680834 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7q86\" (UniqueName: \"kubernetes.io/projected/af279baa-9613-456a-8870-6fd9f306eaaa-kube-api-access-f7q86\") pod \"af279baa-9613-456a-8870-6fd9f306eaaa\" (UID: \"af279baa-9613-456a-8870-6fd9f306eaaa\") " Jul 17 18:12:44.681209 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:44.680915 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af279baa-9613-456a-8870-6fd9f306eaaa-openshift-service-ca\") pod \"af279baa-9613-456a-8870-6fd9f306eaaa\" (UID: \"af279baa-9613-456a-8870-6fd9f306eaaa\") " Jul 17 18:12:44.681209 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:44.681004 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af279baa-9613-456a-8870-6fd9f306eaaa-maas-api-tls\") pod \"af279baa-9613-456a-8870-6fd9f306eaaa\" (UID: \"af279baa-9613-456a-8870-6fd9f306eaaa\") " Jul 17 18:12:44.684952 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:44.684647 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af279baa-9613-456a-8870-6fd9f306eaaa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af279baa-9613-456a-8870-6fd9f306eaaa" (UID: "af279baa-9613-456a-8870-6fd9f306eaaa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:12:44.686177 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:44.685377 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af279baa-9613-456a-8870-6fd9f306eaaa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af279baa-9613-456a-8870-6fd9f306eaaa" (UID: "af279baa-9613-456a-8870-6fd9f306eaaa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:12:44.688109 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:44.688075 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af279baa-9613-456a-8870-6fd9f306eaaa-kube-api-access-f7q86" (OuterVolumeSpecName: "kube-api-access-f7q86") pod "af279baa-9613-456a-8870-6fd9f306eaaa" (UID: "af279baa-9613-456a-8870-6fd9f306eaaa"). InnerVolumeSpecName "kube-api-access-f7q86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:12:44.782929 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:44.782895 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af279baa-9613-456a-8870-6fd9f306eaaa-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:12:44.782929 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:44.782926 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7q86\" (UniqueName: \"kubernetes.io/projected/af279baa-9613-456a-8870-6fd9f306eaaa-kube-api-access-f7q86\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:12:44.783129 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:44.782937 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af279baa-9613-456a-8870-6fd9f306eaaa-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:12:45.518098 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:45.518061 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" event={"ID":"af279baa-9613-456a-8870-6fd9f306eaaa","Type":"ContainerDied","Data":"f5ecbf5df48070f93be89122a08972202c3d01c95106c091cf77a5d28d4130cd"} Jul 17 18:12:45.518098 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:45.518086 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5" Jul 17 18:12:45.518576 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:45.518126 2573 scope.go:117] "RemoveContainer" containerID="2b248289f4a0821c5c621dfbd4252b4e3a485a6ffdce594dc61ffae79ebf3a90" Jul 17 18:12:45.539815 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:45.539775 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5"] Jul 17 18:12:45.545957 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:45.545937 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c50cdb-778cc68ccd-nsjp5"] Jul 17 18:12:45.777412 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:45.777327 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af279baa-9613-456a-8870-6fd9f306eaaa" path="/var/lib/kubelet/pods/af279baa-9613-456a-8870-6fd9f306eaaa/volumes" Jul 17 18:12:49.027298 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:49.027263 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:55.091826 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:55.091789 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:55.191074 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:55.191034 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:59.781606 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:59.781573 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2"] Jul 17 18:12:59.782020 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:59.781794 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" podUID="b97187d2-c474-4e5d-9418-b32d4a37a148" containerName="istio-proxy" containerID="cri-o://3ccf7087f8ec02bd5dd650a618582d205145c780866edcf31db75af6040f9fdd" gracePeriod=30 Jul 17 18:12:59.822067 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:59.822041 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:59.985278 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:12:59.985245 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:00.347249 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.347214 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:00.367500 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.367463 2573 patch_prober.go:28] interesting pod/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 18:13:00.367639 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.367510 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" podUID="b97187d2-c474-4e5d-9418-b32d4a37a148" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 18:13:00.639875 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.639790 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47"] Jul 17 18:13:00.640203 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.640189 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af279baa-9613-456a-8870-6fd9f306eaaa" containerName="maas-api" Jul 17 18:13:00.640203 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.640203 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="af279baa-9613-456a-8870-6fd9f306eaaa" containerName="maas-api" Jul 17 18:13:00.640332 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.640212 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="337abab0-7def-4cb3-b2ef-baf8e9350b26" containerName="istio-proxy" Jul 17 18:13:00.640332 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.640219 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="337abab0-7def-4cb3-b2ef-baf8e9350b26" containerName="istio-proxy" Jul 17 18:13:00.640332 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.640227 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e8656e8-5c3b-4809-8366-99ead858ed26" containerName="maas-api" Jul 17 18:13:00.640332 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.640234 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e8656e8-5c3b-4809-8366-99ead858ed26" containerName="maas-api" Jul 17 18:13:00.640332 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.640244 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="337abab0-7def-4cb3-b2ef-baf8e9350b26" containerName="istio-proxy" Jul 17 18:13:00.640332 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.640250 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="337abab0-7def-4cb3-b2ef-baf8e9350b26" containerName="istio-proxy" Jul 17 18:13:00.640332 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.640331 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="af279baa-9613-456a-8870-6fd9f306eaaa" containerName="maas-api" Jul 17 18:13:00.640684 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.640345 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e8656e8-5c3b-4809-8366-99ead858ed26" containerName="maas-api" Jul 17 18:13:00.640684 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.640354 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="337abab0-7def-4cb3-b2ef-baf8e9350b26" containerName="istio-proxy" Jul 17 18:13:00.640684 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.640368 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="337abab0-7def-4cb3-b2ef-baf8e9350b26" containerName="istio-proxy" Jul 17 18:13:00.643885 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.643866 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.646493 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.646472 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a27826-openshift-default-dockercfg-m97jk\"" Jul 17 18:13:00.658580 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.658557 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47"] Jul 17 18:13:00.669811 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.669780 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:00.738752 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.738714 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-data\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.738946 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.738760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26clx\" (UniqueName: \"kubernetes.io/projected/697ec887-d821-4aa4-9570-3dddb5d9b920-kube-api-access-26clx\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.738946 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.738791 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-workload-socket\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.738946 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.738816 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-envoy\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.738946 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.738845 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-workload-certs\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.738946 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.738915 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-podinfo\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.739186 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.738963 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-credential-socket\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.739186 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.738989 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/697ec887-d821-4aa4-9570-3dddb5d9b920-istiod-ca-cert\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.739186 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.739017 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-token\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.771098 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.771066 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:00.840551 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.840520 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-podinfo\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.841081 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.840563 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-credential-socket\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.841081 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.840587 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/697ec887-d821-4aa4-9570-3dddb5d9b920-istiod-ca-cert\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.841081 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.840614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-token\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.841081 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.840759 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-data\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.841081 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.840786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26clx\" (UniqueName: \"kubernetes.io/projected/697ec887-d821-4aa4-9570-3dddb5d9b920-kube-api-access-26clx\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.841081 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.840816 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-workload-socket\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.841081 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.840847 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-envoy\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.841081 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.840881 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-workload-certs\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.841081 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.841059 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-credential-socket\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.841479 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.841183 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-data\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.841479 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.841252 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-workload-certs\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.841479 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.841379 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-workload-socket\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.841575 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.841481 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/697ec887-d821-4aa4-9570-3dddb5d9b920-istiod-ca-cert\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.843233 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.843210 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-envoy\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.843388 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.843369 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-podinfo\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.848737 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.848717 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-token\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.849141 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.849123 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26clx\" (UniqueName: \"kubernetes.io/projected/697ec887-d821-4aa4-9570-3dddb5d9b920-kube-api-access-26clx\") pod \"e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:00.954720 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:00.954635 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:01.098391 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:01.098369 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47"] Jul 17 18:13:01.100169 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:13:01.100144 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697ec887_d821_4aa4_9570_3dddb5d9b920.slice/crio-3ab8d22263762469e8d6be729b1fe9fdeca61f48ccac0ae5ed541c6060ffcbe7 WatchSource:0}: Error finding container 3ab8d22263762469e8d6be729b1fe9fdeca61f48ccac0ae5ed541c6060ffcbe7: Status 404 returned error can't find the container with id 3ab8d22263762469e8d6be729b1fe9fdeca61f48ccac0ae5ed541c6060ffcbe7 Jul 17 18:13:01.102733 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:01.102698 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:13:01.102847 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:01.102761 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:13:01.102847 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:01.102824 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:13:01.597857 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:01.597820 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" event={"ID":"697ec887-d821-4aa4-9570-3dddb5d9b920","Type":"ContainerStarted","Data":"6435a34bfd1a1040d190e99fdc678ac983bc828bfd0c00b44d2686e8f7e728bf"} Jul 17 18:13:01.598057 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:01.597875 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" event={"ID":"697ec887-d821-4aa4-9570-3dddb5d9b920","Type":"ContainerStarted","Data":"3ab8d22263762469e8d6be729b1fe9fdeca61f48ccac0ae5ed541c6060ffcbe7"} Jul 17 18:13:01.620348 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:01.620299 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" podStartSLOduration=1.620284536 podStartE2EDuration="1.620284536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:13:01.619470251 +0000 UTC m=+2902.354894301" watchObservedRunningTime="2026-07-17 18:13:01.620284536 +0000 UTC m=+2902.355708585" Jul 17 18:13:01.955213 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:01.955128 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:02.960203 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:02.960173 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:03.535479 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.535439 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:03.607932 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.607893 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:03.608873 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.608851 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:03.643075 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.643042 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:03.777582 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.777556 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d"] Jul 17 18:13:03.781205 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.781182 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" Jul 17 18:13:03.783578 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.783559 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a27826-serving-cert\"" Jul 17 18:13:03.790883 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.790862 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d"] Jul 17 18:13:03.809230 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.809203 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:03.873014 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.872977 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a12e13c-50fc-47f6-a66b-c0f7da913569-openshift-service-ca\") pod \"maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d\" (UID: \"5a12e13c-50fc-47f6-a66b-c0f7da913569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" Jul 17 18:13:03.873211 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.873052 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zqzl\" (UniqueName: \"kubernetes.io/projected/5a12e13c-50fc-47f6-a66b-c0f7da913569-kube-api-access-7zqzl\") pod \"maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d\" (UID: \"5a12e13c-50fc-47f6-a66b-c0f7da913569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" Jul 17 18:13:03.873211 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.873093 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a12e13c-50fc-47f6-a66b-c0f7da913569-maas-api-tls\") pod \"maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d\" (UID: \"5a12e13c-50fc-47f6-a66b-c0f7da913569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" Jul 17 18:13:03.974118 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.974083 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a12e13c-50fc-47f6-a66b-c0f7da913569-openshift-service-ca\") pod \"maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d\" (UID: \"5a12e13c-50fc-47f6-a66b-c0f7da913569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" Jul 17 18:13:03.974118 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.974122 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zqzl\" (UniqueName: \"kubernetes.io/projected/5a12e13c-50fc-47f6-a66b-c0f7da913569-kube-api-access-7zqzl\") pod \"maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d\" (UID: \"5a12e13c-50fc-47f6-a66b-c0f7da913569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" Jul 17 18:13:03.974672 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.974149 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a12e13c-50fc-47f6-a66b-c0f7da913569-maas-api-tls\") pod \"maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d\" (UID: \"5a12e13c-50fc-47f6-a66b-c0f7da913569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" Jul 17 18:13:03.974927 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.974903 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a12e13c-50fc-47f6-a66b-c0f7da913569-openshift-service-ca\") pod \"maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d\" (UID: \"5a12e13c-50fc-47f6-a66b-c0f7da913569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" Jul 17 18:13:03.976614 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.976595 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a12e13c-50fc-47f6-a66b-c0f7da913569-maas-api-tls\") pod \"maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d\" (UID: \"5a12e13c-50fc-47f6-a66b-c0f7da913569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" Jul 17 18:13:03.983802 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.983767 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zqzl\" (UniqueName: \"kubernetes.io/projected/5a12e13c-50fc-47f6-a66b-c0f7da913569-kube-api-access-7zqzl\") pod \"maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d\" (UID: \"5a12e13c-50fc-47f6-a66b-c0f7da913569\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" Jul 17 18:13:03.999961 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:03.999938 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:04.094292 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:04.094217 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" Jul 17 18:13:04.247903 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:04.247875 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d"] Jul 17 18:13:04.249560 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:13:04.249536 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a12e13c_50fc_47f6_a66b_c0f7da913569.slice/crio-edbd644f1e986ace9cd0ac610f7504c49024210654accab47951b25a30bd3869 WatchSource:0}: Error finding container edbd644f1e986ace9cd0ac610f7504c49024210654accab47951b25a30bd3869: Status 404 returned error can't find the container with id edbd644f1e986ace9cd0ac610f7504c49024210654accab47951b25a30bd3869 Jul 17 18:13:04.251045 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:04.251027 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:13:04.349286 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:04.349193 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:04.445880 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:04.445843 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:04.614752 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:04.614599 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" event={"ID":"5a12e13c-50fc-47f6-a66b-c0f7da913569","Type":"ContainerStarted","Data":"edbd644f1e986ace9cd0ac610f7504c49024210654accab47951b25a30bd3869"} Jul 17 18:13:05.069298 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.069273 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:13:05.186778 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.186699 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-workload-certs\") pod \"b97187d2-c474-4e5d-9418-b32d4a37a148\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " Jul 17 18:13:05.186778 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.186773 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97187d2-c474-4e5d-9418-b32d4a37a148-istiod-ca-cert\") pod \"b97187d2-c474-4e5d-9418-b32d4a37a148\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " Jul 17 18:13:05.187000 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.186805 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-credential-socket\") pod \"b97187d2-c474-4e5d-9418-b32d4a37a148\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " Jul 17 18:13:05.187000 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.186832 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-token\") pod \"b97187d2-c474-4e5d-9418-b32d4a37a148\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " Jul 17 18:13:05.187000 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.186865 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-workload-socket\") pod \"b97187d2-c474-4e5d-9418-b32d4a37a148\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " Jul 17 18:13:05.187000 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.186889 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9rcg\" (UniqueName: \"kubernetes.io/projected/b97187d2-c474-4e5d-9418-b32d4a37a148-kube-api-access-r9rcg\") pod \"b97187d2-c474-4e5d-9418-b32d4a37a148\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " Jul 17 18:13:05.187000 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.186943 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-data\") pod \"b97187d2-c474-4e5d-9418-b32d4a37a148\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " Jul 17 18:13:05.187000 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.186969 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-envoy\") pod \"b97187d2-c474-4e5d-9418-b32d4a37a148\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " Jul 17 18:13:05.187000 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.186970 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b97187d2-c474-4e5d-9418-b32d4a37a148" (UID: "b97187d2-c474-4e5d-9418-b32d4a37a148"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:13:05.187355 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.187051 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-podinfo\") pod \"b97187d2-c474-4e5d-9418-b32d4a37a148\" (UID: \"b97187d2-c474-4e5d-9418-b32d4a37a148\") " Jul 17 18:13:05.187355 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.187115 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b97187d2-c474-4e5d-9418-b32d4a37a148" (UID: "b97187d2-c474-4e5d-9418-b32d4a37a148"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:13:05.187355 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.187130 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b97187d2-c474-4e5d-9418-b32d4a37a148" (UID: "b97187d2-c474-4e5d-9418-b32d4a37a148"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:13:05.187355 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.187304 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97187d2-c474-4e5d-9418-b32d4a37a148-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b97187d2-c474-4e5d-9418-b32d4a37a148" (UID: "b97187d2-c474-4e5d-9418-b32d4a37a148"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:13:05.187557 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.187455 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:05.187557 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.187472 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97187d2-c474-4e5d-9418-b32d4a37a148-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:05.187557 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.187485 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:05.187557 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.187497 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:05.187908 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.187884 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-data" (OuterVolumeSpecName: "istio-data") pod "b97187d2-c474-4e5d-9418-b32d4a37a148" (UID: "b97187d2-c474-4e5d-9418-b32d4a37a148"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:13:05.189156 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.189135 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97187d2-c474-4e5d-9418-b32d4a37a148-kube-api-access-r9rcg" (OuterVolumeSpecName: "kube-api-access-r9rcg") pod "b97187d2-c474-4e5d-9418-b32d4a37a148" (UID: "b97187d2-c474-4e5d-9418-b32d4a37a148"). InnerVolumeSpecName "kube-api-access-r9rcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:13:05.189156 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.189150 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-token" (OuterVolumeSpecName: "istio-token") pod "b97187d2-c474-4e5d-9418-b32d4a37a148" (UID: "b97187d2-c474-4e5d-9418-b32d4a37a148"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:13:05.189593 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.189571 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b97187d2-c474-4e5d-9418-b32d4a37a148" (UID: "b97187d2-c474-4e5d-9418-b32d4a37a148"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:13:05.189593 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.189581 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b97187d2-c474-4e5d-9418-b32d4a37a148" (UID: "b97187d2-c474-4e5d-9418-b32d4a37a148"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 18:13:05.288400 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.288373 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9rcg\" (UniqueName: \"kubernetes.io/projected/b97187d2-c474-4e5d-9418-b32d4a37a148-kube-api-access-r9rcg\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:05.288400 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.288399 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:05.288400 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.288407 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:05.288643 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.288415 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:05.288643 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.288423 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97187d2-c474-4e5d-9418-b32d4a37a148-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:05.621218 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.621186 2573 generic.go:356] "Generic (PLEG): container finished" podID="b97187d2-c474-4e5d-9418-b32d4a37a148" containerID="3ccf7087f8ec02bd5dd650a618582d205145c780866edcf31db75af6040f9fdd" exitCode=0 Jul 17 18:13:05.621418 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.621223 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" event={"ID":"b97187d2-c474-4e5d-9418-b32d4a37a148","Type":"ContainerDied","Data":"3ccf7087f8ec02bd5dd650a618582d205145c780866edcf31db75af6040f9fdd"} Jul 17 18:13:05.621418 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.621247 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" Jul 17 18:13:05.621418 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.621284 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2" event={"ID":"b97187d2-c474-4e5d-9418-b32d4a37a148","Type":"ContainerDied","Data":"ff2f8c74b5940e9775eecbdcd53969907697e24e1218a35122555e8a921011bd"} Jul 17 18:13:05.621418 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.621315 2573 scope.go:117] "RemoveContainer" containerID="3ccf7087f8ec02bd5dd650a618582d205145c780866edcf31db75af6040f9fdd" Jul 17 18:13:05.622959 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.622933 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" event={"ID":"5a12e13c-50fc-47f6-a66b-c0f7da913569","Type":"ContainerStarted","Data":"a9b93bda80fb90e3804b85d7bcc01aa4dd3d38482a82608752d74c2cc0ec0362"} Jul 17 18:13:05.623103 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.622983 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" Jul 17 18:13:05.630511 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.630496 2573 scope.go:117] "RemoveContainer" containerID="321729487f770e3fceb6d369f0f9ad390cb3e6886ca72935699089dd06807d45" Jul 17 18:13:05.638212 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.638195 2573 scope.go:117] "RemoveContainer" containerID="3ccf7087f8ec02bd5dd650a618582d205145c780866edcf31db75af6040f9fdd" Jul 17 18:13:05.638478 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:13:05.638460 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ccf7087f8ec02bd5dd650a618582d205145c780866edcf31db75af6040f9fdd\": container with ID starting with 3ccf7087f8ec02bd5dd650a618582d205145c780866edcf31db75af6040f9fdd not found: ID does not exist" containerID="3ccf7087f8ec02bd5dd650a618582d205145c780866edcf31db75af6040f9fdd" Jul 17 18:13:05.638534 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.638487 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ccf7087f8ec02bd5dd650a618582d205145c780866edcf31db75af6040f9fdd"} err="failed to get container status \"3ccf7087f8ec02bd5dd650a618582d205145c780866edcf31db75af6040f9fdd\": rpc error: code = NotFound desc = could not find container \"3ccf7087f8ec02bd5dd650a618582d205145c780866edcf31db75af6040f9fdd\": container with ID starting with 3ccf7087f8ec02bd5dd650a618582d205145c780866edcf31db75af6040f9fdd not found: ID does not exist" Jul 17 18:13:05.638534 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.638503 2573 scope.go:117] "RemoveContainer" containerID="321729487f770e3fceb6d369f0f9ad390cb3e6886ca72935699089dd06807d45" Jul 17 18:13:05.638749 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:13:05.638728 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321729487f770e3fceb6d369f0f9ad390cb3e6886ca72935699089dd06807d45\": container with ID starting with 321729487f770e3fceb6d369f0f9ad390cb3e6886ca72935699089dd06807d45 not found: ID does not exist" containerID="321729487f770e3fceb6d369f0f9ad390cb3e6886ca72935699089dd06807d45" Jul 17 18:13:05.638816 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.638758 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321729487f770e3fceb6d369f0f9ad390cb3e6886ca72935699089dd06807d45"} err="failed to get container status \"321729487f770e3fceb6d369f0f9ad390cb3e6886ca72935699089dd06807d45\": rpc error: code = NotFound desc = could not find container \"321729487f770e3fceb6d369f0f9ad390cb3e6886ca72935699089dd06807d45\": container with ID starting with 321729487f770e3fceb6d369f0f9ad390cb3e6886ca72935699089dd06807d45 not found: ID does not exist" Jul 17 18:13:05.660331 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.660310 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2"] Jul 17 18:13:05.667426 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.667406 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c50cdb-openshift-default-5d95677dd9-qmdf2"] Jul 17 18:13:05.704321 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.704282 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" podStartSLOduration=2.104778877 podStartE2EDuration="2.704268409s" totalImagesPullingTime="599.489532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:13:05.703504016 +0000 UTC m=+2906.438928074" watchObservedRunningTime="2026-07-17 18:13:05.704268409 +0000 UTC m=+2906.439692458" Jul 17 18:13:05.778370 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:05.778339 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b97187d2-c474-4e5d-9418-b32d4a37a148" path="/var/lib/kubelet/pods/b97187d2-c474-4e5d-9418-b32d4a37a148/volumes" Jul 17 18:13:08.028929 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:08.028889 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:08.501726 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:08.501695 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:08.836176 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:08.836104 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w"] Jul 17 18:13:08.836584 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:08.836571 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97187d2-c474-4e5d-9418-b32d4a37a148" containerName="istio-proxy" Jul 17 18:13:08.836637 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:08.836586 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97187d2-c474-4e5d-9418-b32d4a37a148" containerName="istio-proxy" Jul 17 18:13:08.836722 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:08.836710 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97187d2-c474-4e5d-9418-b32d4a37a148" containerName="istio-proxy" Jul 17 18:13:08.836759 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:08.836723 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97187d2-c474-4e5d-9418-b32d4a37a148" containerName="istio-proxy" Jul 17 18:13:08.841262 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:08.841244 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:08.846305 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:08.846286 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5ebcf6-openshift-default-dockercfg-6dfwp\"" Jul 17 18:13:08.881082 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:08.881057 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w"] Jul 17 18:13:08.926172 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:08.926143 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:09.025041 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.025010 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-podinfo\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.025229 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.025057 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-data\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.025229 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.025102 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-token\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.025229 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.025121 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-envoy\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.025229 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.025157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-workload-socket\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.025229 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.025209 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-workload-certs\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.025396 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.025260 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-credential-socket\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.025396 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.025283 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istiod-ca-cert\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.025396 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.025303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fgh2\" (UniqueName: \"kubernetes.io/projected/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-kube-api-access-8fgh2\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.126096 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.126014 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-data\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.126096 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.126060 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-token\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.126541 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.126153 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-envoy\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.126541 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.126199 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-workload-socket\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.126541 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.126224 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-workload-certs\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.126541 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.126268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-credential-socket\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.126541 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.126298 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istiod-ca-cert\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.126541 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.126328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fgh2\" (UniqueName: \"kubernetes.io/projected/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-kube-api-access-8fgh2\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.126541 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.126388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-podinfo\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.126541 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.126432 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-data\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.126896 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.126572 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-workload-socket\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.126896 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.126671 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-workload-certs\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.126896 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.126710 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-credential-socket\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.127040 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.127022 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istiod-ca-cert\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.128225 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.128209 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-envoy\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.128910 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.128893 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-podinfo\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.134994 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.134968 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-token\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.135204 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.135186 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fgh2\" (UniqueName: \"kubernetes.io/projected/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-kube-api-access-8fgh2\") pod \"e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.151183 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.151156 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:09.216529 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.216475 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:09.295652 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.295613 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w"] Jul 17 18:13:09.296938 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:13:09.296911 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ecfc3c_7ac6_45b7_a2dc_f5c7210ba5a4.slice/crio-e463d18ffe6aa6eb4caf9bc776fe0c026b3417fb04ddffee1710d3a8c3632a3b WatchSource:0}: Error finding container e463d18ffe6aa6eb4caf9bc776fe0c026b3417fb04ddffee1710d3a8c3632a3b: Status 404 returned error can't find the container with id e463d18ffe6aa6eb4caf9bc776fe0c026b3417fb04ddffee1710d3a8c3632a3b Jul 17 18:13:09.299298 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.299263 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:13:09.299392 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.299345 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:13:09.299463 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.299389 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:13:09.647666 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.647604 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" event={"ID":"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4","Type":"ContainerStarted","Data":"196644f9d0e5d9ad6158d821efad6ceef1c0fc1a91ab07452d826672d5d4a337"} Jul 17 18:13:09.647929 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.647800 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" event={"ID":"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4","Type":"ContainerStarted","Data":"e463d18ffe6aa6eb4caf9bc776fe0c026b3417fb04ddffee1710d3a8c3632a3b"} Jul 17 18:13:09.674344 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:09.674292 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" podStartSLOduration=1.674276332 podStartE2EDuration="1.674276332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:13:09.673964824 +0000 UTC m=+2910.409388885" watchObservedRunningTime="2026-07-17 18:13:09.674276332 +0000 UTC m=+2910.409700381" Jul 17 18:13:10.151724 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:10.151685 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:10.156713 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:10.156680 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:10.653180 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:10.653146 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:10.654088 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:10.654059 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:11.634652 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:11.634603 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" Jul 17 18:13:11.728643 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:11.724642 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:11.816651 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:11.816068 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:11.827643 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:11.820733 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn"] Jul 17 18:13:11.827643 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:11.821394 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97187d2-c474-4e5d-9418-b32d4a37a148" containerName="istio-proxy" Jul 17 18:13:11.827643 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:11.821411 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97187d2-c474-4e5d-9418-b32d4a37a148" containerName="istio-proxy" Jul 17 18:13:11.833058 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:11.833030 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn"] Jul 17 18:13:11.833363 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:11.833303 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" Jul 17 18:13:11.839408 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:11.839388 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5ebcf6-serving-cert\"" Jul 17 18:13:11.895296 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:11.895230 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:11.959298 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:11.959260 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563e5089-e391-4409-a53d-fe442a91830d-maas-api-tls\") pod \"maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn\" (UID: \"563e5089-e391-4409-a53d-fe442a91830d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" Jul 17 18:13:11.959496 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:11.959312 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563e5089-e391-4409-a53d-fe442a91830d-openshift-service-ca\") pod \"maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn\" (UID: \"563e5089-e391-4409-a53d-fe442a91830d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" Jul 17 18:13:11.959496 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:11.959389 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpfd7\" (UniqueName: \"kubernetes.io/projected/563e5089-e391-4409-a53d-fe442a91830d-kube-api-access-vpfd7\") pod \"maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn\" (UID: \"563e5089-e391-4409-a53d-fe442a91830d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" Jul 17 18:13:12.060954 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:12.060901 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563e5089-e391-4409-a53d-fe442a91830d-maas-api-tls\") pod \"maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn\" (UID: \"563e5089-e391-4409-a53d-fe442a91830d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" Jul 17 18:13:12.061165 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:12.060986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563e5089-e391-4409-a53d-fe442a91830d-openshift-service-ca\") pod \"maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn\" (UID: \"563e5089-e391-4409-a53d-fe442a91830d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" Jul 17 18:13:12.061165 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:12.061019 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpfd7\" (UniqueName: \"kubernetes.io/projected/563e5089-e391-4409-a53d-fe442a91830d-kube-api-access-vpfd7\") pod \"maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn\" (UID: \"563e5089-e391-4409-a53d-fe442a91830d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" Jul 17 18:13:12.061165 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:13:12.061069 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-serving-cert: secret "maas-api-e2e-api-b-5ebcf6-serving-cert" not found Jul 17 18:13:12.061165 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:13:12.061130 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/563e5089-e391-4409-a53d-fe442a91830d-maas-api-tls podName:563e5089-e391-4409-a53d-fe442a91830d nodeName:}" failed. No retries permitted until 2026-07-17 18:13:12.561113276 +0000 UTC m=+2913.296537303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/563e5089-e391-4409-a53d-fe442a91830d-maas-api-tls") pod "maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" (UID: "563e5089-e391-4409-a53d-fe442a91830d") : secret "maas-api-e2e-api-b-5ebcf6-serving-cert" not found Jul 17 18:13:12.061647 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:12.061598 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563e5089-e391-4409-a53d-fe442a91830d-openshift-service-ca\") pod \"maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn\" (UID: \"563e5089-e391-4409-a53d-fe442a91830d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" Jul 17 18:13:12.075310 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:12.075286 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpfd7\" (UniqueName: \"kubernetes.io/projected/563e5089-e391-4409-a53d-fe442a91830d-kube-api-access-vpfd7\") pod \"maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn\" (UID: \"563e5089-e391-4409-a53d-fe442a91830d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" Jul 17 18:13:12.209289 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:12.209191 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:12.566204 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:12.566174 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563e5089-e391-4409-a53d-fe442a91830d-maas-api-tls\") pod \"maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn\" (UID: \"563e5089-e391-4409-a53d-fe442a91830d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" Jul 17 18:13:12.568600 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:12.568577 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563e5089-e391-4409-a53d-fe442a91830d-maas-api-tls\") pod \"maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn\" (UID: \"563e5089-e391-4409-a53d-fe442a91830d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" Jul 17 18:13:12.681353 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:12.681317 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:12.746144 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:12.746109 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" Jul 17 18:13:12.783808 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:12.783772 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:12.930986 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:12.930773 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn"] Jul 17 18:13:12.935679 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:13:12.935657 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563e5089_e391_4409_a53d_fe442a91830d.slice/crio-3e517bf1a72fd979b36085308ca970ed4cad116f15a75a0bceb2ce397594d242 WatchSource:0}: Error finding container 3e517bf1a72fd979b36085308ca970ed4cad116f15a75a0bceb2ce397594d242: Status 404 returned error can't find the container with id 3e517bf1a72fd979b36085308ca970ed4cad116f15a75a0bceb2ce397594d242 Jul 17 18:13:13.672419 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:13.672331 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" event={"ID":"563e5089-e391-4409-a53d-fe442a91830d","Type":"ContainerStarted","Data":"4127e89ec61ec217e927c72f23cd15237172187fc1b59565041a4beaea1c0731"} Jul 17 18:13:13.672419 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:13.672413 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" event={"ID":"563e5089-e391-4409-a53d-fe442a91830d","Type":"ContainerStarted","Data":"3e517bf1a72fd979b36085308ca970ed4cad116f15a75a0bceb2ce397594d242"} Jul 17 18:13:13.672580 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:13.672510 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" Jul 17 18:13:13.691760 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:13.691689 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" podStartSLOduration=2.325610423 podStartE2EDuration="2.691674497s" totalImagesPullingTime="366.064074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:13:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:13:13.691259858 +0000 UTC m=+2914.426683904" watchObservedRunningTime="2026-07-17 18:13:13.691674497 +0000 UTC m=+2914.427098546" Jul 17 18:13:16.138700 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:16.138607 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:18.609575 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:18.609491 2573 patch_prober.go:28] interesting pod/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 17 18:13:18.609575 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:18.609549 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" podUID="697ec887-d821-4aa4-9570-3dddb5d9b920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 17 18:13:18.704392 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:18.704359 2573 generic.go:356] "Generic (PLEG): container finished" podID="697ec887-d821-4aa4-9570-3dddb5d9b920" containerID="6435a34bfd1a1040d190e99fdc678ac983bc828bfd0c00b44d2686e8f7e728bf" exitCode=0 Jul 17 18:13:18.704552 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:18.704398 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" event={"ID":"697ec887-d821-4aa4-9570-3dddb5d9b920","Type":"ContainerDied","Data":"6435a34bfd1a1040d190e99fdc678ac983bc828bfd0c00b44d2686e8f7e728bf"} Jul 17 18:13:18.704885 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:18.704869 2573 scope.go:117] "RemoveContainer" containerID="6435a34bfd1a1040d190e99fdc678ac983bc828bfd0c00b44d2686e8f7e728bf" Jul 17 18:13:18.706014 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:18.705974 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:13:18.706098 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:18.706070 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:13:18.706158 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:18.706142 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:13:19.685998 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:19.685968 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" Jul 17 18:13:19.716800 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:19.716736 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" event={"ID":"697ec887-d821-4aa4-9570-3dddb5d9b920","Type":"ContainerStarted","Data":"62980f5db3979906ae15b7829c2c3aa537f5c056c4a8b078c4deb3c8179dcea1"} Jul 17 18:13:19.955550 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:19.955464 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:19.955872 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:19.955765 2573 patch_prober.go:28] interesting pod/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 17 18:13:19.955993 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:19.955901 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" podUID="697ec887-d821-4aa4-9570-3dddb5d9b920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 17 18:13:20.955039 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:20.954999 2573 patch_prober.go:28] interesting pod/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 17 18:13:20.955435 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:20.955054 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" podUID="697ec887-d821-4aa4-9570-3dddb5d9b920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 17 18:13:21.955667 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:21.955602 2573 patch_prober.go:28] interesting pod/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 17 18:13:21.956143 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:21.955695 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" podUID="697ec887-d821-4aa4-9570-3dddb5d9b920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 17 18:13:22.955885 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:22.955846 2573 patch_prober.go:28] interesting pod/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 17 18:13:22.956272 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:22.955897 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" podUID="697ec887-d821-4aa4-9570-3dddb5d9b920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 17 18:13:23.959315 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:23.959284 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:23.959765 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:23.959649 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:23.960390 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:23.960363 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:24.766769 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:24.766732 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:24.870264 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:24.870161 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:26.694821 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:26.694774 2573 patch_prober.go:28] interesting pod/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 17 18:13:26.695412 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:26.694853 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" podUID="49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 17 18:13:27.771163 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:27.771130 2573 generic.go:356] "Generic (PLEG): container finished" podID="49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" containerID="196644f9d0e5d9ad6158d821efad6ceef1c0fc1a91ab07452d826672d5d4a337" exitCode=0 Jul 17 18:13:27.776395 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:27.776371 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" event={"ID":"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4","Type":"ContainerDied","Data":"196644f9d0e5d9ad6158d821efad6ceef1c0fc1a91ab07452d826672d5d4a337"} Jul 17 18:13:27.776802 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:27.776786 2573 scope.go:117] "RemoveContainer" containerID="196644f9d0e5d9ad6158d821efad6ceef1c0fc1a91ab07452d826672d5d4a337" Jul 17 18:13:27.777868 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:27.777839 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:13:27.777937 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:27.777923 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:13:27.777980 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:27.777965 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:13:28.777726 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:28.777695 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" event={"ID":"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4","Type":"ContainerStarted","Data":"c7da521cf2b524028ac7f22c13d3dba88d6455affde30133519b1884eb155448"} Jul 17 18:13:29.152154 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:29.152069 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:29.152303 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:29.152285 2573 patch_prober.go:28] interesting pod/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 17 18:13:29.152345 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:29.152330 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" podUID="49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 17 18:13:30.151905 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:30.151868 2573 patch_prober.go:28] interesting pod/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 17 18:13:30.152288 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:30.151940 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" podUID="49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 17 18:13:31.151993 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:31.151948 2573 patch_prober.go:28] interesting pod/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 17 18:13:31.152473 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:31.152041 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" podUID="49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 17 18:13:32.152073 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:32.152031 2573 patch_prober.go:28] interesting pod/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 17 18:13:32.152469 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:32.152101 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" podUID="49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 17 18:13:33.155601 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:33.155567 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:33.156125 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:33.155852 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:33.156488 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:33.156470 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:13:33.501937 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:33.501905 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:33.603865 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:33.603832 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:34.420114 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.420084 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d"] Jul 17 18:13:34.420499 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.420276 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" podUID="5a12e13c-50fc-47f6-a66b-c0f7da913569" containerName="maas-api" containerID="cri-o://a9b93bda80fb90e3804b85d7bcc01aa4dd3d38482a82608752d74c2cc0ec0362" gracePeriod=30 Jul 17 18:13:34.473765 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.473712 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:34.669954 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.669930 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" Jul 17 18:13:34.785133 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.785098 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a12e13c-50fc-47f6-a66b-c0f7da913569-maas-api-tls\") pod \"5a12e13c-50fc-47f6-a66b-c0f7da913569\" (UID: \"5a12e13c-50fc-47f6-a66b-c0f7da913569\") " Jul 17 18:13:34.785312 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.785174 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zqzl\" (UniqueName: \"kubernetes.io/projected/5a12e13c-50fc-47f6-a66b-c0f7da913569-kube-api-access-7zqzl\") pod \"5a12e13c-50fc-47f6-a66b-c0f7da913569\" (UID: \"5a12e13c-50fc-47f6-a66b-c0f7da913569\") " Jul 17 18:13:34.785312 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.785202 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a12e13c-50fc-47f6-a66b-c0f7da913569-openshift-service-ca\") pod \"5a12e13c-50fc-47f6-a66b-c0f7da913569\" (UID: \"5a12e13c-50fc-47f6-a66b-c0f7da913569\") " Jul 17 18:13:34.785574 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.785551 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a12e13c-50fc-47f6-a66b-c0f7da913569-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5a12e13c-50fc-47f6-a66b-c0f7da913569" (UID: "5a12e13c-50fc-47f6-a66b-c0f7da913569"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:13:34.787157 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.787136 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a12e13c-50fc-47f6-a66b-c0f7da913569-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5a12e13c-50fc-47f6-a66b-c0f7da913569" (UID: "5a12e13c-50fc-47f6-a66b-c0f7da913569"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:13:34.787235 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.787136 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a12e13c-50fc-47f6-a66b-c0f7da913569-kube-api-access-7zqzl" (OuterVolumeSpecName: "kube-api-access-7zqzl") pod "5a12e13c-50fc-47f6-a66b-c0f7da913569" (UID: "5a12e13c-50fc-47f6-a66b-c0f7da913569"). InnerVolumeSpecName "kube-api-access-7zqzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:13:34.810264 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.810237 2573 generic.go:356] "Generic (PLEG): container finished" podID="5a12e13c-50fc-47f6-a66b-c0f7da913569" containerID="a9b93bda80fb90e3804b85d7bcc01aa4dd3d38482a82608752d74c2cc0ec0362" exitCode=0 Jul 17 18:13:34.810380 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.810278 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" Jul 17 18:13:34.810380 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.810321 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" event={"ID":"5a12e13c-50fc-47f6-a66b-c0f7da913569","Type":"ContainerDied","Data":"a9b93bda80fb90e3804b85d7bcc01aa4dd3d38482a82608752d74c2cc0ec0362"} Jul 17 18:13:34.810490 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.810378 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d" event={"ID":"5a12e13c-50fc-47f6-a66b-c0f7da913569","Type":"ContainerDied","Data":"edbd644f1e986ace9cd0ac610f7504c49024210654accab47951b25a30bd3869"} Jul 17 18:13:34.810490 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.810410 2573 scope.go:117] "RemoveContainer" containerID="a9b93bda80fb90e3804b85d7bcc01aa4dd3d38482a82608752d74c2cc0ec0362" Jul 17 18:13:34.819687 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.819670 2573 scope.go:117] "RemoveContainer" containerID="a9b93bda80fb90e3804b85d7bcc01aa4dd3d38482a82608752d74c2cc0ec0362" Jul 17 18:13:34.819939 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:13:34.819922 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b93bda80fb90e3804b85d7bcc01aa4dd3d38482a82608752d74c2cc0ec0362\": container with ID starting with a9b93bda80fb90e3804b85d7bcc01aa4dd3d38482a82608752d74c2cc0ec0362 not found: ID does not exist" containerID="a9b93bda80fb90e3804b85d7bcc01aa4dd3d38482a82608752d74c2cc0ec0362" Jul 17 18:13:34.820006 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.819946 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b93bda80fb90e3804b85d7bcc01aa4dd3d38482a82608752d74c2cc0ec0362"} err="failed to get container status \"a9b93bda80fb90e3804b85d7bcc01aa4dd3d38482a82608752d74c2cc0ec0362\": rpc error: code = NotFound desc = could not find container \"a9b93bda80fb90e3804b85d7bcc01aa4dd3d38482a82608752d74c2cc0ec0362\": container with ID starting with a9b93bda80fb90e3804b85d7bcc01aa4dd3d38482a82608752d74c2cc0ec0362 not found: ID does not exist" Jul 17 18:13:34.833677 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.833645 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d"] Jul 17 18:13:34.837211 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.837190 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a27826-58476bd6bc-9xj6d"] Jul 17 18:13:34.887329 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.887306 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a12e13c-50fc-47f6-a66b-c0f7da913569-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:34.887437 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.887333 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zqzl\" (UniqueName: \"kubernetes.io/projected/5a12e13c-50fc-47f6-a66b-c0f7da913569-kube-api-access-7zqzl\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:34.887437 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:34.887349 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a12e13c-50fc-47f6-a66b-c0f7da913569-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:35.778277 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:35.778239 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a12e13c-50fc-47f6-a66b-c0f7da913569" path="/var/lib/kubelet/pods/5a12e13c-50fc-47f6-a66b-c0f7da913569/volumes" Jul 17 18:13:39.319420 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:39.319382 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:45.358831 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:45.358800 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:45.464507 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:45.464468 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:49.992786 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:49.992749 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47"] Jul 17 18:13:49.993280 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:49.992943 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" podUID="697ec887-d821-4aa4-9570-3dddb5d9b920" containerName="istio-proxy" containerID="cri-o://62980f5db3979906ae15b7829c2c3aa537f5c056c4a8b078c4deb3c8179dcea1" gracePeriod=30 Jul 17 18:13:50.014679 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:50.014652 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:50.246143 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:50.246058 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:53.961989 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:53.961955 2573 patch_prober.go:28] interesting pod/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 18:13:53.962360 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:53.962024 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" podUID="697ec887-d821-4aa4-9570-3dddb5d9b920" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 18:13:55.228235 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.228206 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:55.390862 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.390771 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-credential-socket\") pod \"697ec887-d821-4aa4-9570-3dddb5d9b920\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " Jul 17 18:13:55.390862 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.390818 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-workload-socket\") pod \"697ec887-d821-4aa4-9570-3dddb5d9b920\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " Jul 17 18:13:55.390862 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.390846 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-workload-certs\") pod \"697ec887-d821-4aa4-9570-3dddb5d9b920\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " Jul 17 18:13:55.391154 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.390874 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-token\") pod \"697ec887-d821-4aa4-9570-3dddb5d9b920\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " Jul 17 18:13:55.391154 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.390932 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-podinfo\") pod \"697ec887-d821-4aa4-9570-3dddb5d9b920\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " Jul 17 18:13:55.391154 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.390957 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26clx\" (UniqueName: \"kubernetes.io/projected/697ec887-d821-4aa4-9570-3dddb5d9b920-kube-api-access-26clx\") pod \"697ec887-d821-4aa4-9570-3dddb5d9b920\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " Jul 17 18:13:55.391154 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.390973 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-envoy\") pod \"697ec887-d821-4aa4-9570-3dddb5d9b920\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " Jul 17 18:13:55.391154 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.391029 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-data\") pod \"697ec887-d821-4aa4-9570-3dddb5d9b920\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " Jul 17 18:13:55.391154 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.391101 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "697ec887-d821-4aa4-9570-3dddb5d9b920" (UID: "697ec887-d821-4aa4-9570-3dddb5d9b920"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:13:55.391414 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.391191 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/697ec887-d821-4aa4-9570-3dddb5d9b920-istiod-ca-cert\") pod \"697ec887-d821-4aa4-9570-3dddb5d9b920\" (UID: \"697ec887-d821-4aa4-9570-3dddb5d9b920\") " Jul 17 18:13:55.391414 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.391197 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "697ec887-d821-4aa4-9570-3dddb5d9b920" (UID: "697ec887-d821-4aa4-9570-3dddb5d9b920"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:13:55.391414 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.391200 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "697ec887-d821-4aa4-9570-3dddb5d9b920" (UID: "697ec887-d821-4aa4-9570-3dddb5d9b920"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:13:55.391602 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.391571 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/697ec887-d821-4aa4-9570-3dddb5d9b920-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "697ec887-d821-4aa4-9570-3dddb5d9b920" (UID: "697ec887-d821-4aa4-9570-3dddb5d9b920"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:13:55.391699 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.391584 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:55.391699 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.391653 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:55.391699 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.391676 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:55.391964 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.391943 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-data" (OuterVolumeSpecName: "istio-data") pod "697ec887-d821-4aa4-9570-3dddb5d9b920" (UID: "697ec887-d821-4aa4-9570-3dddb5d9b920"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:13:55.393425 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.393393 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "697ec887-d821-4aa4-9570-3dddb5d9b920" (UID: "697ec887-d821-4aa4-9570-3dddb5d9b920"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 18:13:55.393557 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.393439 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697ec887-d821-4aa4-9570-3dddb5d9b920-kube-api-access-26clx" (OuterVolumeSpecName: "kube-api-access-26clx") pod "697ec887-d821-4aa4-9570-3dddb5d9b920" (UID: "697ec887-d821-4aa4-9570-3dddb5d9b920"). InnerVolumeSpecName "kube-api-access-26clx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:13:55.393735 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.393708 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-token" (OuterVolumeSpecName: "istio-token") pod "697ec887-d821-4aa4-9570-3dddb5d9b920" (UID: "697ec887-d821-4aa4-9570-3dddb5d9b920"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:13:55.393807 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.393732 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "697ec887-d821-4aa4-9570-3dddb5d9b920" (UID: "697ec887-d821-4aa4-9570-3dddb5d9b920"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:13:55.492559 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.492528 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:55.492559 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.492555 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:55.492763 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.492565 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26clx\" (UniqueName: \"kubernetes.io/projected/697ec887-d821-4aa4-9570-3dddb5d9b920-kube-api-access-26clx\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:55.492763 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.492576 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:55.492763 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.492583 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/697ec887-d821-4aa4-9570-3dddb5d9b920-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:55.492763 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.492591 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/697ec887-d821-4aa4-9570-3dddb5d9b920-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:13:55.919346 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.919303 2573 generic.go:356] "Generic (PLEG): container finished" podID="697ec887-d821-4aa4-9570-3dddb5d9b920" containerID="62980f5db3979906ae15b7829c2c3aa537f5c056c4a8b078c4deb3c8179dcea1" exitCode=0 Jul 17 18:13:55.919537 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.919354 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" Jul 17 18:13:55.919537 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.919359 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" event={"ID":"697ec887-d821-4aa4-9570-3dddb5d9b920","Type":"ContainerDied","Data":"62980f5db3979906ae15b7829c2c3aa537f5c056c4a8b078c4deb3c8179dcea1"} Jul 17 18:13:55.919537 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.919416 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47" event={"ID":"697ec887-d821-4aa4-9570-3dddb5d9b920","Type":"ContainerDied","Data":"3ab8d22263762469e8d6be729b1fe9fdeca61f48ccac0ae5ed541c6060ffcbe7"} Jul 17 18:13:55.919537 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.919444 2573 scope.go:117] "RemoveContainer" containerID="62980f5db3979906ae15b7829c2c3aa537f5c056c4a8b078c4deb3c8179dcea1" Jul 17 18:13:55.928017 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.927999 2573 scope.go:117] "RemoveContainer" containerID="6435a34bfd1a1040d190e99fdc678ac983bc828bfd0c00b44d2686e8f7e728bf" Jul 17 18:13:55.935630 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.935602 2573 scope.go:117] "RemoveContainer" containerID="62980f5db3979906ae15b7829c2c3aa537f5c056c4a8b078c4deb3c8179dcea1" Jul 17 18:13:55.935971 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:13:55.935948 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62980f5db3979906ae15b7829c2c3aa537f5c056c4a8b078c4deb3c8179dcea1\": container with ID starting with 62980f5db3979906ae15b7829c2c3aa537f5c056c4a8b078c4deb3c8179dcea1 not found: ID does not exist" containerID="62980f5db3979906ae15b7829c2c3aa537f5c056c4a8b078c4deb3c8179dcea1" Jul 17 18:13:55.936068 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.935978 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62980f5db3979906ae15b7829c2c3aa537f5c056c4a8b078c4deb3c8179dcea1"} err="failed to get container status \"62980f5db3979906ae15b7829c2c3aa537f5c056c4a8b078c4deb3c8179dcea1\": rpc error: code = NotFound desc = could not find container \"62980f5db3979906ae15b7829c2c3aa537f5c056c4a8b078c4deb3c8179dcea1\": container with ID starting with 62980f5db3979906ae15b7829c2c3aa537f5c056c4a8b078c4deb3c8179dcea1 not found: ID does not exist" Jul 17 18:13:55.936068 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.935996 2573 scope.go:117] "RemoveContainer" containerID="6435a34bfd1a1040d190e99fdc678ac983bc828bfd0c00b44d2686e8f7e728bf" Jul 17 18:13:55.936238 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:13:55.936220 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6435a34bfd1a1040d190e99fdc678ac983bc828bfd0c00b44d2686e8f7e728bf\": container with ID starting with 6435a34bfd1a1040d190e99fdc678ac983bc828bfd0c00b44d2686e8f7e728bf not found: ID does not exist" containerID="6435a34bfd1a1040d190e99fdc678ac983bc828bfd0c00b44d2686e8f7e728bf" Jul 17 18:13:55.936281 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.936245 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6435a34bfd1a1040d190e99fdc678ac983bc828bfd0c00b44d2686e8f7e728bf"} err="failed to get container status \"6435a34bfd1a1040d190e99fdc678ac983bc828bfd0c00b44d2686e8f7e728bf\": rpc error: code = NotFound desc = could not find container \"6435a34bfd1a1040d190e99fdc678ac983bc828bfd0c00b44d2686e8f7e728bf\": container with ID starting with 6435a34bfd1a1040d190e99fdc678ac983bc828bfd0c00b44d2686e8f7e728bf not found: ID does not exist" Jul 17 18:13:55.948025 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.947994 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47"] Jul 17 18:13:55.953658 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:55.953635 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a27826-openshift-default-c6c9f485f-6dg47"] Jul 17 18:13:57.778073 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:13:57.778039 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="697ec887-d821-4aa4-9570-3dddb5d9b920" path="/var/lib/kubelet/pods/697ec887-d821-4aa4-9570-3dddb5d9b920/volumes" Jul 17 18:14:00.816758 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:00.816668 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn"] Jul 17 18:14:00.817197 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:00.816936 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" podUID="563e5089-e391-4409-a53d-fe442a91830d" containerName="maas-api" containerID="cri-o://4127e89ec61ec217e927c72f23cd15237172187fc1b59565041a4beaea1c0731" gracePeriod=30 Jul 17 18:14:00.849965 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:00.849940 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:00.947095 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:00.947066 2573 generic.go:356] "Generic (PLEG): container finished" podID="563e5089-e391-4409-a53d-fe442a91830d" containerID="4127e89ec61ec217e927c72f23cd15237172187fc1b59565041a4beaea1c0731" exitCode=0 Jul 17 18:14:00.947219 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:00.947110 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" event={"ID":"563e5089-e391-4409-a53d-fe442a91830d","Type":"ContainerDied","Data":"4127e89ec61ec217e927c72f23cd15237172187fc1b59565041a4beaea1c0731"} Jul 17 18:14:01.068009 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:01.067953 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" Jul 17 18:14:01.145100 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:01.145072 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpfd7\" (UniqueName: \"kubernetes.io/projected/563e5089-e391-4409-a53d-fe442a91830d-kube-api-access-vpfd7\") pod \"563e5089-e391-4409-a53d-fe442a91830d\" (UID: \"563e5089-e391-4409-a53d-fe442a91830d\") " Jul 17 18:14:01.145268 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:01.145111 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563e5089-e391-4409-a53d-fe442a91830d-maas-api-tls\") pod \"563e5089-e391-4409-a53d-fe442a91830d\" (UID: \"563e5089-e391-4409-a53d-fe442a91830d\") " Jul 17 18:14:01.145268 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:01.145151 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563e5089-e391-4409-a53d-fe442a91830d-openshift-service-ca\") pod \"563e5089-e391-4409-a53d-fe442a91830d\" (UID: \"563e5089-e391-4409-a53d-fe442a91830d\") " Jul 17 18:14:01.145578 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:01.145555 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/563e5089-e391-4409-a53d-fe442a91830d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "563e5089-e391-4409-a53d-fe442a91830d" (UID: "563e5089-e391-4409-a53d-fe442a91830d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:14:01.147224 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:01.147194 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563e5089-e391-4409-a53d-fe442a91830d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "563e5089-e391-4409-a53d-fe442a91830d" (UID: "563e5089-e391-4409-a53d-fe442a91830d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:14:01.147296 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:01.147222 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563e5089-e391-4409-a53d-fe442a91830d-kube-api-access-vpfd7" (OuterVolumeSpecName: "kube-api-access-vpfd7") pod "563e5089-e391-4409-a53d-fe442a91830d" (UID: "563e5089-e391-4409-a53d-fe442a91830d"). InnerVolumeSpecName "kube-api-access-vpfd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:14:01.245981 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:01.245957 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563e5089-e391-4409-a53d-fe442a91830d-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:14:01.245981 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:01.245982 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpfd7\" (UniqueName: \"kubernetes.io/projected/563e5089-e391-4409-a53d-fe442a91830d-kube-api-access-vpfd7\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:14:01.246127 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:01.245992 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563e5089-e391-4409-a53d-fe442a91830d-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:14:01.953351 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:01.953310 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" event={"ID":"563e5089-e391-4409-a53d-fe442a91830d","Type":"ContainerDied","Data":"3e517bf1a72fd979b36085308ca970ed4cad116f15a75a0bceb2ce397594d242"} Jul 17 18:14:01.953846 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:01.953363 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn" Jul 17 18:14:01.953846 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:01.953373 2573 scope.go:117] "RemoveContainer" containerID="4127e89ec61ec217e927c72f23cd15237172187fc1b59565041a4beaea1c0731" Jul 17 18:14:01.970767 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:01.970742 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn"] Jul 17 18:14:01.972540 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:01.972520 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5ebcf6-69fb9fd54f-6c2wn"] Jul 17 18:14:03.777330 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:03.777295 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="563e5089-e391-4409-a53d-fe442a91830d" path="/var/lib/kubelet/pods/563e5089-e391-4409-a53d-fe442a91830d/volumes" Jul 17 18:14:05.453603 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:05.453566 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:11.264945 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:11.264914 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:11.339766 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:11.339734 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:16.161922 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:16.161879 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w"] Jul 17 18:14:16.162383 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:16.162164 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" podUID="49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" containerName="istio-proxy" containerID="cri-o://c7da521cf2b524028ac7f22c13d3dba88d6455affde30133519b1884eb155448" gracePeriod=30 Jul 17 18:14:16.193970 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:16.193939 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:16.427056 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:16.426978 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:16.800325 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:16.800292 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:17.141746 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.141668 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s"] Jul 17 18:14:17.142083 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.142070 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a12e13c-50fc-47f6-a66b-c0f7da913569" containerName="maas-api" Jul 17 18:14:17.142125 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.142084 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a12e13c-50fc-47f6-a66b-c0f7da913569" containerName="maas-api" Jul 17 18:14:17.142125 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.142101 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="697ec887-d821-4aa4-9570-3dddb5d9b920" containerName="istio-proxy" Jul 17 18:14:17.142125 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.142107 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="697ec887-d821-4aa4-9570-3dddb5d9b920" containerName="istio-proxy" Jul 17 18:14:17.142125 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.142122 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563e5089-e391-4409-a53d-fe442a91830d" containerName="maas-api" Jul 17 18:14:17.142250 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.142128 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="563e5089-e391-4409-a53d-fe442a91830d" containerName="maas-api" Jul 17 18:14:17.142250 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.142143 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="697ec887-d821-4aa4-9570-3dddb5d9b920" containerName="istio-proxy" Jul 17 18:14:17.142250 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.142148 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="697ec887-d821-4aa4-9570-3dddb5d9b920" containerName="istio-proxy" Jul 17 18:14:17.142250 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.142201 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="697ec887-d821-4aa4-9570-3dddb5d9b920" containerName="istio-proxy" Jul 17 18:14:17.142250 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.142209 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="697ec887-d821-4aa4-9570-3dddb5d9b920" containerName="istio-proxy" Jul 17 18:14:17.142250 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.142215 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="563e5089-e391-4409-a53d-fe442a91830d" containerName="maas-api" Jul 17 18:14:17.142250 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.142224 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a12e13c-50fc-47f6-a66b-c0f7da913569" containerName="maas-api" Jul 17 18:14:17.145149 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.145131 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.150412 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.150391 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-4699ff-openshift-default-dockercfg-859m2\"" Jul 17 18:14:17.198568 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.198540 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s"] Jul 17 18:14:17.203662 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.203607 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-workload-socket\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.203779 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.203692 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25qzd\" (UniqueName: \"kubernetes.io/projected/5897f210-3e09-484b-af11-042588e0a77d-kube-api-access-25qzd\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.203779 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.203721 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-istio-envoy\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.203779 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.203744 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-istio-data\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.203779 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.203766 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5897f210-3e09-484b-af11-042588e0a77d-istio-podinfo\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.203919 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.203794 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5897f210-3e09-484b-af11-042588e0a77d-istio-token\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.203919 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.203848 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-workload-certs\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.203919 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.203888 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-credential-socket\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.204015 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.203957 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5897f210-3e09-484b-af11-042588e0a77d-istiod-ca-cert\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.295115 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.295079 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:17.304814 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.304785 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5897f210-3e09-484b-af11-042588e0a77d-istiod-ca-cert\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.304978 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.304850 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-workload-socket\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.304978 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.304904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25qzd\" (UniqueName: \"kubernetes.io/projected/5897f210-3e09-484b-af11-042588e0a77d-kube-api-access-25qzd\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.304978 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.304931 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-istio-envoy\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.304978 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.304976 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-istio-data\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.305314 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.305236 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-workload-socket\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.305437 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.305001 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5897f210-3e09-484b-af11-042588e0a77d-istio-podinfo\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.305437 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.305377 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5897f210-3e09-484b-af11-042588e0a77d-istio-token\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.305437 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.305416 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-workload-certs\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.305598 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.305462 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-credential-socket\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.305598 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.305481 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-istio-data\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.305598 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.305576 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5897f210-3e09-484b-af11-042588e0a77d-istiod-ca-cert\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.305847 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.305787 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-credential-socket\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.307235 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.307189 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-workload-certs\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.310127 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.310105 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5897f210-3e09-484b-af11-042588e0a77d-istio-podinfo\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.310338 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.310294 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-istio-envoy\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.316086 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.316060 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25qzd\" (UniqueName: \"kubernetes.io/projected/5897f210-3e09-484b-af11-042588e0a77d-kube-api-access-25qzd\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.317380 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.317359 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5897f210-3e09-484b-af11-042588e0a77d-istio-token\") pod \"e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.415246 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.415160 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:17.455405 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.455371 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:17.593093 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.593062 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s"] Jul 17 18:14:17.595785 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:14:17.595761 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5897f210_3e09_484b_af11_042588e0a77d.slice/crio-185a664a3329e20768c5b725bb7ab87a13fc64a805ed4d82c6d3c5e91223adce WatchSource:0}: Error finding container 185a664a3329e20768c5b725bb7ab87a13fc64a805ed4d82c6d3c5e91223adce: Status 404 returned error can't find the container with id 185a664a3329e20768c5b725bb7ab87a13fc64a805ed4d82c6d3c5e91223adce Jul 17 18:14:17.598061 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.598029 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:14:17.598141 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.598101 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:14:17.598141 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:17.598129 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:14:18.035264 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:18.035228 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" event={"ID":"5897f210-3e09-484b-af11-042588e0a77d","Type":"ContainerStarted","Data":"30bf574212c6948b6ab24cbf9b57be56296b7b2dc62d8357e045410e97d668c1"} Jul 17 18:14:18.035447 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:18.035279 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" event={"ID":"5897f210-3e09-484b-af11-042588e0a77d","Type":"ContainerStarted","Data":"185a664a3329e20768c5b725bb7ab87a13fc64a805ed4d82c6d3c5e91223adce"} Jul 17 18:14:18.104369 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:18.104314 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" podStartSLOduration=1.104297137 podStartE2EDuration="1.104297137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:14:18.102815611 +0000 UTC m=+2978.838239663" watchObservedRunningTime="2026-07-17 18:14:18.104297137 +0000 UTC m=+2978.839721187" Jul 17 18:14:18.158141 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:18.158109 2573 patch_prober.go:28] interesting pod/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 18:14:18.158309 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:18.158160 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" podUID="49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 18:14:18.456154 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:18.456061 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:18.460888 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:18.460863 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:19.040334 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:19.040296 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:19.041423 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:19.041403 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:20.041240 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.041201 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:20.134850 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.134810 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:20.177508 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.177466 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs"] Jul 17 18:14:20.180917 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.180902 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" Jul 17 18:14:20.183751 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.183730 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-4699ff-serving-cert\"" Jul 17 18:14:20.228191 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.228161 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs"] Jul 17 18:14:20.336020 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.335943 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a82b571-9098-44f9-afd2-1ef573a4781c-maas-api-tls\") pod \"maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs\" (UID: \"2a82b571-9098-44f9-afd2-1ef573a4781c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" Jul 17 18:14:20.336020 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.336000 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d47kr\" (UniqueName: \"kubernetes.io/projected/2a82b571-9098-44f9-afd2-1ef573a4781c-kube-api-access-d47kr\") pod \"maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs\" (UID: \"2a82b571-9098-44f9-afd2-1ef573a4781c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" Jul 17 18:14:20.336188 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.336072 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a82b571-9098-44f9-afd2-1ef573a4781c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs\" (UID: \"2a82b571-9098-44f9-afd2-1ef573a4781c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" Jul 17 18:14:20.344468 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.344436 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:20.436877 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.436840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a82b571-9098-44f9-afd2-1ef573a4781c-maas-api-tls\") pod \"maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs\" (UID: \"2a82b571-9098-44f9-afd2-1ef573a4781c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" Jul 17 18:14:20.437043 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.436916 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d47kr\" (UniqueName: \"kubernetes.io/projected/2a82b571-9098-44f9-afd2-1ef573a4781c-kube-api-access-d47kr\") pod \"maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs\" (UID: \"2a82b571-9098-44f9-afd2-1ef573a4781c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" Jul 17 18:14:20.437043 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.436958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a82b571-9098-44f9-afd2-1ef573a4781c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs\" (UID: \"2a82b571-9098-44f9-afd2-1ef573a4781c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" Jul 17 18:14:20.437043 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:14:20.436993 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-serving-cert: secret "maas-api-e2e-shared-a-4699ff-serving-cert" not found Jul 17 18:14:20.437219 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:14:20.437076 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a82b571-9098-44f9-afd2-1ef573a4781c-maas-api-tls podName:2a82b571-9098-44f9-afd2-1ef573a4781c nodeName:}" failed. No retries permitted until 2026-07-17 18:14:20.937056949 +0000 UTC m=+2981.672480982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a82b571-9098-44f9-afd2-1ef573a4781c-maas-api-tls") pod "maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" (UID: "2a82b571-9098-44f9-afd2-1ef573a4781c") : secret "maas-api-e2e-shared-a-4699ff-serving-cert" not found Jul 17 18:14:20.437669 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.437644 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a82b571-9098-44f9-afd2-1ef573a4781c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs\" (UID: \"2a82b571-9098-44f9-afd2-1ef573a4781c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" Jul 17 18:14:20.449400 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.449374 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d47kr\" (UniqueName: \"kubernetes.io/projected/2a82b571-9098-44f9-afd2-1ef573a4781c-kube-api-access-d47kr\") pod \"maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs\" (UID: \"2a82b571-9098-44f9-afd2-1ef573a4781c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" Jul 17 18:14:20.683936 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.683852 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:20.836033 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.836000 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:20.937740 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.937662 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:20.941905 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.941879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a82b571-9098-44f9-afd2-1ef573a4781c-maas-api-tls\") pod \"maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs\" (UID: \"2a82b571-9098-44f9-afd2-1ef573a4781c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" Jul 17 18:14:20.944360 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:20.944331 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a82b571-9098-44f9-afd2-1ef573a4781c-maas-api-tls\") pod \"maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs\" (UID: \"2a82b571-9098-44f9-afd2-1ef573a4781c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" Jul 17 18:14:21.091582 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.091546 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" Jul 17 18:14:21.227952 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.227867 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs"] Jul 17 18:14:21.232638 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:14:21.232596 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a82b571_9098_44f9_afd2_1ef573a4781c.slice/crio-0f0033cf9122d63f4cbe34c15ddb29dca61a7dff15c8eea5187190a9d42ae830 WatchSource:0}: Error finding container 0f0033cf9122d63f4cbe34c15ddb29dca61a7dff15c8eea5187190a9d42ae830: Status 404 returned error can't find the container with id 0f0033cf9122d63f4cbe34c15ddb29dca61a7dff15c8eea5187190a9d42ae830 Jul 17 18:14:21.410176 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.410149 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:14:21.548764 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.548734 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istiod-ca-cert\") pod \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " Jul 17 18:14:21.548922 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.548786 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-podinfo\") pod \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " Jul 17 18:14:21.548922 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.548821 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-workload-certs\") pod \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " Jul 17 18:14:21.548922 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.548852 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-token\") pod \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " Jul 17 18:14:21.548922 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.548877 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-workload-socket\") pod \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " Jul 17 18:14:21.548922 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.548907 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fgh2\" (UniqueName: \"kubernetes.io/projected/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-kube-api-access-8fgh2\") pod \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " Jul 17 18:14:21.549176 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.548942 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-data\") pod \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " Jul 17 18:14:21.549176 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.548969 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-envoy\") pod \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " Jul 17 18:14:21.549176 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.549073 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-credential-socket\") pod \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\" (UID: \"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4\") " Jul 17 18:14:21.549324 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.549165 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" (UID: "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:21.549324 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.549180 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" (UID: "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:21.549324 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.549245 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" (UID: "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:14:21.549489 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.549370 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" (UID: "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:21.549540 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.549513 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:14:21.549540 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.549529 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:14:21.549670 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.549541 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:14:21.549670 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.549555 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:14:21.550019 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.549985 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" (UID: "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:21.551301 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.551272 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" (UID: "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:14:21.551301 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.551296 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" (UID: "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 18:14:21.551484 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.551295 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-kube-api-access-8fgh2" (OuterVolumeSpecName: "kube-api-access-8fgh2") pod "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" (UID: "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4"). InnerVolumeSpecName "kube-api-access-8fgh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:14:21.551484 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.551331 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" (UID: "49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:21.650963 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.650931 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:14:21.650963 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.650958 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:14:21.650963 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.650969 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fgh2\" (UniqueName: \"kubernetes.io/projected/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-kube-api-access-8fgh2\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:14:21.651169 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.650977 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:14:21.651169 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:21.650986 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:14:22.057715 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:22.057668 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" event={"ID":"2a82b571-9098-44f9-afd2-1ef573a4781c","Type":"ContainerStarted","Data":"5e3761155962ed29d5db65fd9a78d49aa1a4185824ae34d5a3c4ea9764a371e3"} Jul 17 18:14:22.057924 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:22.057737 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" Jul 17 18:14:22.057924 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:22.057765 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" event={"ID":"2a82b571-9098-44f9-afd2-1ef573a4781c","Type":"ContainerStarted","Data":"0f0033cf9122d63f4cbe34c15ddb29dca61a7dff15c8eea5187190a9d42ae830"} Jul 17 18:14:22.059279 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:22.059251 2573 generic.go:356] "Generic (PLEG): container finished" podID="49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" containerID="c7da521cf2b524028ac7f22c13d3dba88d6455affde30133519b1884eb155448" exitCode=0 Jul 17 18:14:22.059367 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:22.059295 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" Jul 17 18:14:22.059367 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:22.059334 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" event={"ID":"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4","Type":"ContainerDied","Data":"c7da521cf2b524028ac7f22c13d3dba88d6455affde30133519b1884eb155448"} Jul 17 18:14:22.059431 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:22.059374 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w" event={"ID":"49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4","Type":"ContainerDied","Data":"e463d18ffe6aa6eb4caf9bc776fe0c026b3417fb04ddffee1710d3a8c3632a3b"} Jul 17 18:14:22.059431 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:22.059398 2573 scope.go:117] "RemoveContainer" containerID="c7da521cf2b524028ac7f22c13d3dba88d6455affde30133519b1884eb155448" Jul 17 18:14:22.068056 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:22.068024 2573 scope.go:117] "RemoveContainer" containerID="196644f9d0e5d9ad6158d821efad6ceef1c0fc1a91ab07452d826672d5d4a337" Jul 17 18:14:22.075687 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:22.075653 2573 scope.go:117] "RemoveContainer" containerID="c7da521cf2b524028ac7f22c13d3dba88d6455affde30133519b1884eb155448" Jul 17 18:14:22.076758 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:14:22.076606 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7da521cf2b524028ac7f22c13d3dba88d6455affde30133519b1884eb155448\": container with ID starting with c7da521cf2b524028ac7f22c13d3dba88d6455affde30133519b1884eb155448 not found: ID does not exist" containerID="c7da521cf2b524028ac7f22c13d3dba88d6455affde30133519b1884eb155448" Jul 17 18:14:22.076758 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:22.076691 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7da521cf2b524028ac7f22c13d3dba88d6455affde30133519b1884eb155448"} err="failed to get container status \"c7da521cf2b524028ac7f22c13d3dba88d6455affde30133519b1884eb155448\": rpc error: code = NotFound desc = could not find container \"c7da521cf2b524028ac7f22c13d3dba88d6455affde30133519b1884eb155448\": container with ID starting with c7da521cf2b524028ac7f22c13d3dba88d6455affde30133519b1884eb155448 not found: ID does not exist" Jul 17 18:14:22.076919 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:22.076790 2573 scope.go:117] "RemoveContainer" containerID="196644f9d0e5d9ad6158d821efad6ceef1c0fc1a91ab07452d826672d5d4a337" Jul 17 18:14:22.077222 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:14:22.077192 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196644f9d0e5d9ad6158d821efad6ceef1c0fc1a91ab07452d826672d5d4a337\": container with ID starting with 196644f9d0e5d9ad6158d821efad6ceef1c0fc1a91ab07452d826672d5d4a337 not found: ID does not exist" containerID="196644f9d0e5d9ad6158d821efad6ceef1c0fc1a91ab07452d826672d5d4a337" Jul 17 18:14:22.077290 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:22.077233 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196644f9d0e5d9ad6158d821efad6ceef1c0fc1a91ab07452d826672d5d4a337"} err="failed to get container status \"196644f9d0e5d9ad6158d821efad6ceef1c0fc1a91ab07452d826672d5d4a337\": rpc error: code = NotFound desc = could not find container \"196644f9d0e5d9ad6158d821efad6ceef1c0fc1a91ab07452d826672d5d4a337\": container with ID starting with 196644f9d0e5d9ad6158d821efad6ceef1c0fc1a91ab07452d826672d5d4a337 not found: ID does not exist" Jul 17 18:14:22.079073 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:22.079022 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" podStartSLOduration=1.685361828 podStartE2EDuration="2.079004834s" totalImagesPullingTime="393.643006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:14:22.07503084 +0000 UTC m=+2982.810454911" watchObservedRunningTime="2026-07-17 18:14:22.079004834 +0000 UTC m=+2982.814428885" Jul 17 18:14:22.093309 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:22.093287 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w"] Jul 17 18:14:22.098302 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:22.098280 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5ebcf6-openshift-default-69b785dd7f-sbt9w"] Jul 17 18:14:23.777774 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:23.777743 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" path="/var/lib/kubelet/pods/49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4/volumes" Jul 17 18:14:24.545023 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:24.544993 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:24.868822 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:24.868732 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:25.187779 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.187706 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc"] Jul 17 18:14:25.188148 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.188137 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" containerName="istio-proxy" Jul 17 18:14:25.188191 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.188150 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" containerName="istio-proxy" Jul 17 18:14:25.188191 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.188158 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" containerName="istio-proxy" Jul 17 18:14:25.188191 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.188164 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" containerName="istio-proxy" Jul 17 18:14:25.188289 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.188223 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" containerName="istio-proxy" Jul 17 18:14:25.188289 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.188231 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="49ecfc3c-7ac6-45b7-a2dc-f5c7210ba5a4" containerName="istio-proxy" Jul 17 18:14:25.192894 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.192873 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.195539 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.195520 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-22fb10-openshift-default-dockercfg-k7x2f\"" Jul 17 18:14:25.205043 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.205023 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc"] Jul 17 18:14:25.226566 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.226545 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:25.289028 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.288998 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-workload-socket\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.289166 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.289034 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/206d67e3-7157-416a-a92f-22f661cdbe07-istio-podinfo\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.289166 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.289150 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-istio-data\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.289240 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.289185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/206d67e3-7157-416a-a92f-22f661cdbe07-istio-token\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.289274 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.289257 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmnd8\" (UniqueName: \"kubernetes.io/projected/206d67e3-7157-416a-a92f-22f661cdbe07-kube-api-access-rmnd8\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.289314 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.289286 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-credential-socket\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.289345 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.289309 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/206d67e3-7157-416a-a92f-22f661cdbe07-istiod-ca-cert\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.289345 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.289332 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-istio-envoy\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.289403 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.289351 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-workload-certs\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.320852 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.320827 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:25.390043 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.390007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmnd8\" (UniqueName: \"kubernetes.io/projected/206d67e3-7157-416a-a92f-22f661cdbe07-kube-api-access-rmnd8\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.390043 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.390050 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-credential-socket\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.390312 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.390070 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/206d67e3-7157-416a-a92f-22f661cdbe07-istiod-ca-cert\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.390312 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.390093 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-istio-envoy\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.390312 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.390114 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-workload-certs\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.390312 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.390165 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-workload-socket\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.390312 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.390190 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/206d67e3-7157-416a-a92f-22f661cdbe07-istio-podinfo\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.390312 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.390271 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-istio-data\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.390312 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.390307 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/206d67e3-7157-416a-a92f-22f661cdbe07-istio-token\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.390747 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.390527 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-credential-socket\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.390747 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.390703 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-workload-certs\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.390747 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.390712 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-istio-data\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.390894 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.390745 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-workload-socket\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.391033 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.391011 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/206d67e3-7157-416a-a92f-22f661cdbe07-istiod-ca-cert\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.392410 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.392383 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-istio-envoy\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.392652 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.392635 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/206d67e3-7157-416a-a92f-22f661cdbe07-istio-podinfo\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.398808 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.398783 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/206d67e3-7157-416a-a92f-22f661cdbe07-istio-token\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.398968 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.398952 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmnd8\" (UniqueName: \"kubernetes.io/projected/206d67e3-7157-416a-a92f-22f661cdbe07-kube-api-access-rmnd8\") pod \"e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.462051 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.461993 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:25.503437 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.503219 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:25.644918 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.644891 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc"] Jul 17 18:14:25.646926 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:14:25.646892 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod206d67e3_7157_416a_a92f_22f661cdbe07.slice/crio-c9dbd509ca3037ff7ba432601dc13b7f38bf5af29fd5fc4e66de8c0dc2764bbf WatchSource:0}: Error finding container c9dbd509ca3037ff7ba432601dc13b7f38bf5af29fd5fc4e66de8c0dc2764bbf: Status 404 returned error can't find the container with id c9dbd509ca3037ff7ba432601dc13b7f38bf5af29fd5fc4e66de8c0dc2764bbf Jul 17 18:14:25.649132 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.649098 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:14:25.649237 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.649180 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:14:25.649237 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:25.649228 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:14:26.080770 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:26.080730 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" event={"ID":"206d67e3-7157-416a-a92f-22f661cdbe07","Type":"ContainerStarted","Data":"04b6ad6c595a97d0440f17a402e34492861e4ceba9d6314f9d3bfa51b806ad28"} Jul 17 18:14:26.081240 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:26.080784 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" event={"ID":"206d67e3-7157-416a-a92f-22f661cdbe07","Type":"ContainerStarted","Data":"c9dbd509ca3037ff7ba432601dc13b7f38bf5af29fd5fc4e66de8c0dc2764bbf"} Jul 17 18:14:26.101478 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:26.101426 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" podStartSLOduration=1.101412228 podStartE2EDuration="1.101412228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:14:26.099237167 +0000 UTC m=+2986.834661229" watchObservedRunningTime="2026-07-17 18:14:26.101412228 +0000 UTC m=+2986.836836276" Jul 17 18:14:26.504436 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:26.504402 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:26.509317 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:26.509292 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:27.085696 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:27.085661 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:27.086822 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:27.086801 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:27.457895 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:27.457809 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:27.555549 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:27.555515 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:28.071085 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.071049 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" Jul 17 18:14:28.209234 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.209131 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:28.304789 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.304750 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:28.483717 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.483630 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz"] Jul 17 18:14:28.487740 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.487716 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" Jul 17 18:14:28.490516 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.490495 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-22fb10-serving-cert\"" Jul 17 18:14:28.499193 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.499125 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz"] Jul 17 18:14:28.529988 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.529944 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:28.633120 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.633081 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6863a941-0ecf-42b0-8179-52048edb547e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz\" (UID: \"6863a941-0ecf-42b0-8179-52048edb547e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" Jul 17 18:14:28.633374 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.633218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tprhb\" (UniqueName: \"kubernetes.io/projected/6863a941-0ecf-42b0-8179-52048edb547e-kube-api-access-tprhb\") pod \"maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz\" (UID: \"6863a941-0ecf-42b0-8179-52048edb547e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" Jul 17 18:14:28.633374 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.633281 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6863a941-0ecf-42b0-8179-52048edb547e-maas-api-tls\") pod \"maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz\" (UID: \"6863a941-0ecf-42b0-8179-52048edb547e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" Jul 17 18:14:28.733926 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.733884 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tprhb\" (UniqueName: \"kubernetes.io/projected/6863a941-0ecf-42b0-8179-52048edb547e-kube-api-access-tprhb\") pod \"maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz\" (UID: \"6863a941-0ecf-42b0-8179-52048edb547e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" Jul 17 18:14:28.734076 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.733959 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6863a941-0ecf-42b0-8179-52048edb547e-maas-api-tls\") pod \"maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz\" (UID: \"6863a941-0ecf-42b0-8179-52048edb547e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" Jul 17 18:14:28.734076 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.734020 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6863a941-0ecf-42b0-8179-52048edb547e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz\" (UID: \"6863a941-0ecf-42b0-8179-52048edb547e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" Jul 17 18:14:28.734820 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.734791 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6863a941-0ecf-42b0-8179-52048edb547e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz\" (UID: \"6863a941-0ecf-42b0-8179-52048edb547e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" Jul 17 18:14:28.737155 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.737123 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6863a941-0ecf-42b0-8179-52048edb547e-maas-api-tls\") pod \"maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz\" (UID: \"6863a941-0ecf-42b0-8179-52048edb547e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" Jul 17 18:14:28.746711 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.746687 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tprhb\" (UniqueName: \"kubernetes.io/projected/6863a941-0ecf-42b0-8179-52048edb547e-kube-api-access-tprhb\") pod \"maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz\" (UID: \"6863a941-0ecf-42b0-8179-52048edb547e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" Jul 17 18:14:28.811884 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.811833 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" Jul 17 18:14:28.813642 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.813599 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:28.982931 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:28.982901 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz"] Jul 17 18:14:28.984847 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:14:28.984794 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6863a941_0ecf_42b0_8179_52048edb547e.slice/crio-b2423e15a79c67aed7c27be4ab714875e09a06dffe760c171849a554facd1b01 WatchSource:0}: Error finding container b2423e15a79c67aed7c27be4ab714875e09a06dffe760c171849a554facd1b01: Status 404 returned error can't find the container with id b2423e15a79c67aed7c27be4ab714875e09a06dffe760c171849a554facd1b01 Jul 17 18:14:29.098315 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:29.098276 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" event={"ID":"6863a941-0ecf-42b0-8179-52048edb547e","Type":"ContainerStarted","Data":"b2423e15a79c67aed7c27be4ab714875e09a06dffe760c171849a554facd1b01"} Jul 17 18:14:29.156260 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:29.156227 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:29.261459 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:29.261381 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:30.109898 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:30.108917 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" event={"ID":"6863a941-0ecf-42b0-8179-52048edb547e","Type":"ContainerStarted","Data":"134b49d0982d235be081e21da690d2cf3d132db6a74b269d38f0caa557978156"} Jul 17 18:14:30.109898 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:30.109784 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" Jul 17 18:14:30.132176 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:30.132126 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" podStartSLOduration=1.334711284 podStartE2EDuration="2.132112831s" totalImagesPullingTime="797.401547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:14:30.12937194 +0000 UTC m=+2990.864795990" watchObservedRunningTime="2026-07-17 18:14:30.132112831 +0000 UTC m=+2990.867536879" Jul 17 18:14:32.774905 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:32.774870 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:34.042466 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:34.042434 2573 patch_prober.go:28] interesting pod/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 17 18:14:34.042951 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:34.042490 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" podUID="5897f210-3e09-484b-af11-042588e0a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 17 18:14:34.135411 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:34.135375 2573 generic.go:356] "Generic (PLEG): container finished" podID="5897f210-3e09-484b-af11-042588e0a77d" containerID="30bf574212c6948b6ab24cbf9b57be56296b7b2dc62d8357e045410e97d668c1" exitCode=0 Jul 17 18:14:34.135597 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:34.135449 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" event={"ID":"5897f210-3e09-484b-af11-042588e0a77d","Type":"ContainerDied","Data":"30bf574212c6948b6ab24cbf9b57be56296b7b2dc62d8357e045410e97d668c1"} Jul 17 18:14:34.135888 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:34.135872 2573 scope.go:117] "RemoveContainer" containerID="30bf574212c6948b6ab24cbf9b57be56296b7b2dc62d8357e045410e97d668c1" Jul 17 18:14:34.136933 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:34.136907 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:14:34.136995 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:34.136972 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:14:34.137032 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:34.137001 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:14:35.146983 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:35.146935 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" event={"ID":"5897f210-3e09-484b-af11-042588e0a77d","Type":"ContainerStarted","Data":"85fd401dc5772bd9ef7c855726523b7b3725a27d32aa5bd588a8063d2e46972e"} Jul 17 18:14:35.456654 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:35.456548 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:35.456829 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:35.456800 2573 patch_prober.go:28] interesting pod/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 17 18:14:35.456886 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:35.456864 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" podUID="5897f210-3e09-484b-af11-042588e0a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 17 18:14:36.456187 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:36.456150 2573 patch_prober.go:28] interesting pod/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 17 18:14:36.456591 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:36.456225 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" podUID="5897f210-3e09-484b-af11-042588e0a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 17 18:14:37.128076 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:37.128020 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" Jul 17 18:14:37.456313 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:37.456222 2573 patch_prober.go:28] interesting pod/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 17 18:14:37.456313 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:37.456287 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" podUID="5897f210-3e09-484b-af11-042588e0a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 17 18:14:38.456666 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:38.456604 2573 patch_prober.go:28] interesting pod/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 17 18:14:38.457054 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:38.456694 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" podUID="5897f210-3e09-484b-af11-042588e0a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 17 18:14:39.461027 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:39.460989 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:39.461431 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:39.461257 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:39.462023 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:39.462002 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:14:39.952675 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:39.952649 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 18:14:39.956931 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:39.956912 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 18:14:40.181912 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:40.181840 2573 generic.go:356] "Generic (PLEG): container finished" podID="206d67e3-7157-416a-a92f-22f661cdbe07" containerID="04b6ad6c595a97d0440f17a402e34492861e4ceba9d6314f9d3bfa51b806ad28" exitCode=0 Jul 17 18:14:40.182010 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:40.181917 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" event={"ID":"206d67e3-7157-416a-a92f-22f661cdbe07","Type":"ContainerDied","Data":"04b6ad6c595a97d0440f17a402e34492861e4ceba9d6314f9d3bfa51b806ad28"} Jul 17 18:14:40.182401 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:40.182383 2573 scope.go:117] "RemoveContainer" containerID="04b6ad6c595a97d0440f17a402e34492861e4ceba9d6314f9d3bfa51b806ad28" Jul 17 18:14:40.183464 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:40.183436 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:14:40.183564 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:40.183519 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:14:40.183643 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:40.183569 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:14:41.189447 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:41.189408 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" event={"ID":"206d67e3-7157-416a-a92f-22f661cdbe07","Type":"ContainerStarted","Data":"ba9178d55762072d6517c45988cac1fa5d824caacb2ab30c364abd32740e18e5"} Jul 17 18:14:41.503873 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:41.503837 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:41.505458 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:41.505423 2573 patch_prober.go:28] interesting pod/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 17 18:14:41.505586 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:41.505496 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" podUID="206d67e3-7157-416a-a92f-22f661cdbe07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 17 18:14:41.796238 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:41.796131 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:41.890803 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:41.890772 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:42.085958 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:42.085867 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:42.504256 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:42.504212 2573 patch_prober.go:28] interesting pod/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 17 18:14:42.504803 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:42.504279 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" podUID="206d67e3-7157-416a-a92f-22f661cdbe07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 17 18:14:43.504067 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:43.504029 2573 patch_prober.go:28] interesting pod/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 17 18:14:43.504278 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:43.504097 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" podUID="206d67e3-7157-416a-a92f-22f661cdbe07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 17 18:14:44.245594 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:44.245559 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:44.504744 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:44.504636 2573 patch_prober.go:28] interesting pod/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 17 18:14:44.505240 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:44.504743 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" podUID="206d67e3-7157-416a-a92f-22f661cdbe07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 17 18:14:45.507668 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:45.507636 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:45.508636 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:45.508596 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:14:51.129803 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:51.129772 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:51.340202 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:14:51.340171 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:15:07.319805 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:15:07.319768 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:15:38.750990 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:15:38.750950 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:16:40.428616 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:16:40.428575 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:24.297960 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:17:24.297874 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:25.093305 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:17:25.093265 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:27.190949 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:17:27.190920 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:49.333965 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:17:49.333932 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:18:13.219444 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:18:13.219407 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:18:59.310685 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:18:59.310644 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:00.136431 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:19:00.136402 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:10.506349 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:19:10.506313 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:28.389988 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:19:28.389954 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:28.487539 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:19:28.487506 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:29.815474 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:19:29.815429 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:39.984582 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:19:39.984556 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 18:19:39.991419 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:19:39.991402 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7lgg2_6ca760ea-272f-4b74-b3ff-958d46cffced/ovn-acl-logging/0.log" Jul 17 18:19:42.654269 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:19:42.654239 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:42.755393 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:19:42.755350 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:51.481771 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:19:51.481736 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:51.901645 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:19:51.901595 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:52.003128 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:19:52.003096 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:20:13.469420 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:13.469342 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:20:14.205069 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:14.205038 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:20:15.132283 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:15.132246 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7573-140054093700528"] Jul 17 18:20:15.135863 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:15.135846 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-140054093700528" Jul 17 18:20:15.138425 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:15.138407 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hqwf7\"" Jul 17 18:20:15.145108 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:15.145085 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7573-140054093700528"] Jul 17 18:20:15.163035 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:15.163012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2shn\" (UniqueName: \"kubernetes.io/projected/40d7eb2b-6f62-4769-bd70-cef149f40390-kube-api-access-k2shn\") pod \"test-curl-7573-140054093700528\" (UID: \"40d7eb2b-6f62-4769-bd70-cef149f40390\") " pod="opendatahub/test-curl-7573-140054093700528" Jul 17 18:20:15.263969 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:15.263932 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2shn\" (UniqueName: \"kubernetes.io/projected/40d7eb2b-6f62-4769-bd70-cef149f40390-kube-api-access-k2shn\") pod \"test-curl-7573-140054093700528\" (UID: \"40d7eb2b-6f62-4769-bd70-cef149f40390\") " pod="opendatahub/test-curl-7573-140054093700528" Jul 17 18:20:15.276903 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:15.276877 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2shn\" (UniqueName: \"kubernetes.io/projected/40d7eb2b-6f62-4769-bd70-cef149f40390-kube-api-access-k2shn\") pod \"test-curl-7573-140054093700528\" (UID: \"40d7eb2b-6f62-4769-bd70-cef149f40390\") " pod="opendatahub/test-curl-7573-140054093700528" Jul 17 18:20:15.446176 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:15.446087 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-140054093700528" Jul 17 18:20:15.575892 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:15.575854 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7573-140054093700528"] Jul 17 18:20:15.579191 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:20:15.579167 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d7eb2b_6f62_4769_bd70_cef149f40390.slice/crio-99dc618c8688cad19b6c97b598baf651e9ea679fb981054d8f21845091bb33bf WatchSource:0}: Error finding container 99dc618c8688cad19b6c97b598baf651e9ea679fb981054d8f21845091bb33bf: Status 404 returned error can't find the container with id 99dc618c8688cad19b6c97b598baf651e9ea679fb981054d8f21845091bb33bf Jul 17 18:20:15.580376 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:15.580359 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:20:15.925978 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:15.925946 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7573-140054093700528" event={"ID":"40d7eb2b-6f62-4769-bd70-cef149f40390","Type":"ContainerStarted","Data":"99dc618c8688cad19b6c97b598baf651e9ea679fb981054d8f21845091bb33bf"} Jul 17 18:20:16.933129 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:16.933092 2573 generic.go:356] "Generic (PLEG): container finished" podID="40d7eb2b-6f62-4769-bd70-cef149f40390" containerID="3fc0f56e33d4e07f4a58507facf97687d9f60543da37fd7690e18d342f3bdca3" exitCode=0 Jul 17 18:20:16.933495 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:16.933179 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7573-140054093700528" event={"ID":"40d7eb2b-6f62-4769-bd70-cef149f40390","Type":"ContainerDied","Data":"3fc0f56e33d4e07f4a58507facf97687d9f60543da37fd7690e18d342f3bdca3"} Jul 17 18:20:18.071673 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.071647 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-140054093700528" Jul 17 18:20:18.092880 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.092857 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7573-140054093700528_40d7eb2b-6f62-4769-bd70-cef149f40390/test-curl-7573-140054093700528/0.log" Jul 17 18:20:18.113878 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.113848 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7573-140054093700528"] Jul 17 18:20:18.118138 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.118116 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7573-140054093700528"] Jul 17 18:20:18.192045 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.191964 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2shn\" (UniqueName: \"kubernetes.io/projected/40d7eb2b-6f62-4769-bd70-cef149f40390-kube-api-access-k2shn\") pod \"40d7eb2b-6f62-4769-bd70-cef149f40390\" (UID: \"40d7eb2b-6f62-4769-bd70-cef149f40390\") " Jul 17 18:20:18.193970 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.193945 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d7eb2b-6f62-4769-bd70-cef149f40390-kube-api-access-k2shn" (OuterVolumeSpecName: "kube-api-access-k2shn") pod "40d7eb2b-6f62-4769-bd70-cef149f40390" (UID: "40d7eb2b-6f62-4769-bd70-cef149f40390"). InnerVolumeSpecName "kube-api-access-k2shn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:20:18.234764 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.234729 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7573-140054093759024"] Jul 17 18:20:18.235183 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.235168 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d7eb2b-6f62-4769-bd70-cef149f40390" containerName="test-curl-7573-140054093700528" Jul 17 18:20:18.235183 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.235184 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d7eb2b-6f62-4769-bd70-cef149f40390" containerName="test-curl-7573-140054093700528" Jul 17 18:20:18.235276 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.235265 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d7eb2b-6f62-4769-bd70-cef149f40390" containerName="test-curl-7573-140054093700528" Jul 17 18:20:18.238286 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.238268 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-140054093759024" Jul 17 18:20:18.245674 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.245650 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7573-140054093759024"] Jul 17 18:20:18.293328 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.293298 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbgpb\" (UniqueName: \"kubernetes.io/projected/5cd34aec-e3f9-482e-97ad-76a9064dd043-kube-api-access-dbgpb\") pod \"test-curl-7573-140054093759024\" (UID: \"5cd34aec-e3f9-482e-97ad-76a9064dd043\") " pod="opendatahub/test-curl-7573-140054093759024" Jul 17 18:20:18.293486 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.293417 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2shn\" (UniqueName: \"kubernetes.io/projected/40d7eb2b-6f62-4769-bd70-cef149f40390-kube-api-access-k2shn\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:20:18.394882 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.394844 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbgpb\" (UniqueName: \"kubernetes.io/projected/5cd34aec-e3f9-482e-97ad-76a9064dd043-kube-api-access-dbgpb\") pod \"test-curl-7573-140054093759024\" (UID: \"5cd34aec-e3f9-482e-97ad-76a9064dd043\") " pod="opendatahub/test-curl-7573-140054093759024" Jul 17 18:20:18.403689 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.403657 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbgpb\" (UniqueName: \"kubernetes.io/projected/5cd34aec-e3f9-482e-97ad-76a9064dd043-kube-api-access-dbgpb\") pod \"test-curl-7573-140054093759024\" (UID: \"5cd34aec-e3f9-482e-97ad-76a9064dd043\") " pod="opendatahub/test-curl-7573-140054093759024" Jul 17 18:20:18.548016 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.547988 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-140054093759024" Jul 17 18:20:18.677610 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.677583 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7573-140054093759024"] Jul 17 18:20:18.680231 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:20:18.680208 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cd34aec_e3f9_482e_97ad_76a9064dd043.slice/crio-22a8202dae502ba5152198b41eea339f9643db9bddd151224f8df82426222d00 WatchSource:0}: Error finding container 22a8202dae502ba5152198b41eea339f9643db9bddd151224f8df82426222d00: Status 404 returned error can't find the container with id 22a8202dae502ba5152198b41eea339f9643db9bddd151224f8df82426222d00 Jul 17 18:20:18.944921 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.944833 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7573-140054093759024" event={"ID":"5cd34aec-e3f9-482e-97ad-76a9064dd043","Type":"ContainerStarted","Data":"22a8202dae502ba5152198b41eea339f9643db9bddd151224f8df82426222d00"} Jul 17 18:20:18.946043 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.946019 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-140054093700528" Jul 17 18:20:18.946043 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:18.946033 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99dc618c8688cad19b6c97b598baf651e9ea679fb981054d8f21845091bb33bf" Jul 17 18:20:18.957507 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:20:18.957484 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7573-140054093700528\" is forbidden: User \"system:node:ip-10-0-132-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-183.ec2.internal' and this object" podUID="40d7eb2b-6f62-4769-bd70-cef149f40390" pod="opendatahub/test-curl-7573-140054093700528" Jul 17 18:20:19.778004 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:19.777971 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d7eb2b-6f62-4769-bd70-cef149f40390" path="/var/lib/kubelet/pods/40d7eb2b-6f62-4769-bd70-cef149f40390/volumes" Jul 17 18:20:19.778741 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:20:19.778715 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7573-140054093700528\" is forbidden: User \"system:node:ip-10-0-132-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-183.ec2.internal' and this object" podUID="40d7eb2b-6f62-4769-bd70-cef149f40390" pod="opendatahub/test-curl-7573-140054093700528" Jul 17 18:20:19.953103 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:19.953066 2573 generic.go:356] "Generic (PLEG): container finished" podID="5cd34aec-e3f9-482e-97ad-76a9064dd043" containerID="dd7af9ab9016a5763581df11931ecbc03b26b89b25b062d1a14812bd567a8006" exitCode=0 Jul 17 18:20:19.953288 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:19.953158 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7573-140054093759024" event={"ID":"5cd34aec-e3f9-482e-97ad-76a9064dd043","Type":"ContainerDied","Data":"dd7af9ab9016a5763581df11931ecbc03b26b89b25b062d1a14812bd567a8006"} Jul 17 18:20:20.952073 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:20.952041 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:20:21.092555 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:21.092529 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-140054093759024" Jul 17 18:20:21.113231 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:21.113203 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7573-140054093759024_5cd34aec-e3f9-482e-97ad-76a9064dd043/test-curl-7573-140054093759024/0.log" Jul 17 18:20:21.138373 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:21.138345 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7573-140054093759024"] Jul 17 18:20:21.140239 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:21.140214 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7573-140054093759024"] Jul 17 18:20:21.221244 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:21.221171 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbgpb\" (UniqueName: \"kubernetes.io/projected/5cd34aec-e3f9-482e-97ad-76a9064dd043-kube-api-access-dbgpb\") pod \"5cd34aec-e3f9-482e-97ad-76a9064dd043\" (UID: \"5cd34aec-e3f9-482e-97ad-76a9064dd043\") " Jul 17 18:20:21.223189 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:21.223165 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd34aec-e3f9-482e-97ad-76a9064dd043-kube-api-access-dbgpb" (OuterVolumeSpecName: "kube-api-access-dbgpb") pod "5cd34aec-e3f9-482e-97ad-76a9064dd043" (UID: "5cd34aec-e3f9-482e-97ad-76a9064dd043"). InnerVolumeSpecName "kube-api-access-dbgpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:20:21.322881 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:21.322848 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbgpb\" (UniqueName: \"kubernetes.io/projected/5cd34aec-e3f9-482e-97ad-76a9064dd043-kube-api-access-dbgpb\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:20:21.778306 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:21.778270 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cd34aec-e3f9-482e-97ad-76a9064dd043" path="/var/lib/kubelet/pods/5cd34aec-e3f9-482e-97ad-76a9064dd043/volumes" Jul 17 18:20:21.964798 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:21.964766 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-140054093759024" Jul 17 18:20:21.964798 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:21.964792 2573 scope.go:117] "RemoveContainer" containerID="dd7af9ab9016a5763581df11931ecbc03b26b89b25b062d1a14812bd567a8006" Jul 17 18:20:24.495615 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:24.495581 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7573-140054093735232"] Jul 17 18:20:24.496017 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:24.496002 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd34aec-e3f9-482e-97ad-76a9064dd043" containerName="test-curl-7573-140054093759024" Jul 17 18:20:24.496017 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:24.496014 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd34aec-e3f9-482e-97ad-76a9064dd043" containerName="test-curl-7573-140054093759024" Jul 17 18:20:24.496114 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:24.496103 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd34aec-e3f9-482e-97ad-76a9064dd043" containerName="test-curl-7573-140054093759024" Jul 17 18:20:24.500534 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:24.500516 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-140054093735232" Jul 17 18:20:24.503317 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:24.503286 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hqwf7\"" Jul 17 18:20:24.506312 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:24.506285 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7573-140054093735232"] Jul 17 18:20:24.552779 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:24.552752 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtk4h\" (UniqueName: \"kubernetes.io/projected/2e584281-b5fa-46a5-af40-937908f6c380-kube-api-access-rtk4h\") pod \"test-curl-7573-140054093735232\" (UID: \"2e584281-b5fa-46a5-af40-937908f6c380\") " pod="opendatahub/test-curl-7573-140054093735232" Jul 17 18:20:24.654257 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:24.654227 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtk4h\" (UniqueName: \"kubernetes.io/projected/2e584281-b5fa-46a5-af40-937908f6c380-kube-api-access-rtk4h\") pod \"test-curl-7573-140054093735232\" (UID: \"2e584281-b5fa-46a5-af40-937908f6c380\") " pod="opendatahub/test-curl-7573-140054093735232" Jul 17 18:20:24.662338 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:24.662308 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtk4h\" (UniqueName: \"kubernetes.io/projected/2e584281-b5fa-46a5-af40-937908f6c380-kube-api-access-rtk4h\") pod \"test-curl-7573-140054093735232\" (UID: \"2e584281-b5fa-46a5-af40-937908f6c380\") " pod="opendatahub/test-curl-7573-140054093735232" Jul 17 18:20:24.811055 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:24.810977 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-140054093735232" Jul 17 18:20:24.943133 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:24.943101 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7573-140054093735232"] Jul 17 18:20:24.945413 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:20:24.945386 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e584281_b5fa_46a5_af40_937908f6c380.slice/crio-c9999f7e97c615bc0a38d3cff483f34eddddc555c73a92ff9fbfa7174e34b0ee WatchSource:0}: Error finding container c9999f7e97c615bc0a38d3cff483f34eddddc555c73a92ff9fbfa7174e34b0ee: Status 404 returned error can't find the container with id c9999f7e97c615bc0a38d3cff483f34eddddc555c73a92ff9fbfa7174e34b0ee Jul 17 18:20:24.985511 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:24.985486 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7573-140054093735232" event={"ID":"2e584281-b5fa-46a5-af40-937908f6c380","Type":"ContainerStarted","Data":"c9999f7e97c615bc0a38d3cff483f34eddddc555c73a92ff9fbfa7174e34b0ee"} Jul 17 18:20:25.992127 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:25.992089 2573 generic.go:356] "Generic (PLEG): container finished" podID="2e584281-b5fa-46a5-af40-937908f6c380" containerID="27884c8be9bc6c3a3d5761b36c554cac99f2934ec33b5f3a26f0c1039fcea686" exitCode=0 Jul 17 18:20:25.992476 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:25.992173 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7573-140054093735232" event={"ID":"2e584281-b5fa-46a5-af40-937908f6c380","Type":"ContainerDied","Data":"27884c8be9bc6c3a3d5761b36c554cac99f2934ec33b5f3a26f0c1039fcea686"} Jul 17 18:20:27.154985 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:27.154962 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-140054093735232" Jul 17 18:20:27.176861 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:27.176835 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7573-140054093735232_2e584281-b5fa-46a5-af40-937908f6c380/test-curl-7573-140054093735232/0.log" Jul 17 18:20:27.204314 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:27.204291 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7573-140054093735232"] Jul 17 18:20:27.206162 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:27.206144 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7573-140054093735232"] Jul 17 18:20:27.282242 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:27.282165 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtk4h\" (UniqueName: \"kubernetes.io/projected/2e584281-b5fa-46a5-af40-937908f6c380-kube-api-access-rtk4h\") pod \"2e584281-b5fa-46a5-af40-937908f6c380\" (UID: \"2e584281-b5fa-46a5-af40-937908f6c380\") " Jul 17 18:20:27.284213 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:27.284189 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e584281-b5fa-46a5-af40-937908f6c380-kube-api-access-rtk4h" (OuterVolumeSpecName: "kube-api-access-rtk4h") pod "2e584281-b5fa-46a5-af40-937908f6c380" (UID: "2e584281-b5fa-46a5-af40-937908f6c380"). InnerVolumeSpecName "kube-api-access-rtk4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:20:27.383187 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:27.383156 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtk4h\" (UniqueName: \"kubernetes.io/projected/2e584281-b5fa-46a5-af40-937908f6c380-kube-api-access-rtk4h\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:20:27.777890 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:27.777852 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e584281-b5fa-46a5-af40-937908f6c380" path="/var/lib/kubelet/pods/2e584281-b5fa-46a5-af40-937908f6c380/volumes" Jul 17 18:20:28.004566 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:28.004539 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-140054093735232" Jul 17 18:20:28.004566 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:28.004554 2573 scope.go:117] "RemoveContainer" containerID="27884c8be9bc6c3a3d5761b36c554cac99f2934ec33b5f3a26f0c1039fcea686" Jul 17 18:20:33.073287 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:33.073248 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:20:33.175722 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:33.175688 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:20:42.980816 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:42.980781 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs"] Jul 17 18:20:42.981288 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:42.981058 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" podUID="2a82b571-9098-44f9-afd2-1ef573a4781c" containerName="maas-api" containerID="cri-o://5e3761155962ed29d5db65fd9a78d49aa1a4185824ae34d5a3c4ea9764a371e3" gracePeriod=30 Jul 17 18:20:43.011444 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:43.011402 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:20:43.066821 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:43.066777 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" podUID="2a82b571-9098-44f9-afd2-1ef573a4781c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.67:8443/health\": dial tcp 10.134.0.67:8443: connect: connection refused" Jul 17 18:20:43.438222 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:43.438199 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" Jul 17 18:20:43.548255 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:43.548168 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a82b571-9098-44f9-afd2-1ef573a4781c-openshift-service-ca\") pod \"2a82b571-9098-44f9-afd2-1ef573a4781c\" (UID: \"2a82b571-9098-44f9-afd2-1ef573a4781c\") " Jul 17 18:20:43.548435 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:43.548260 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d47kr\" (UniqueName: \"kubernetes.io/projected/2a82b571-9098-44f9-afd2-1ef573a4781c-kube-api-access-d47kr\") pod \"2a82b571-9098-44f9-afd2-1ef573a4781c\" (UID: \"2a82b571-9098-44f9-afd2-1ef573a4781c\") " Jul 17 18:20:43.548435 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:43.548290 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a82b571-9098-44f9-afd2-1ef573a4781c-maas-api-tls\") pod \"2a82b571-9098-44f9-afd2-1ef573a4781c\" (UID: \"2a82b571-9098-44f9-afd2-1ef573a4781c\") " Jul 17 18:20:43.548586 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:43.548556 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a82b571-9098-44f9-afd2-1ef573a4781c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2a82b571-9098-44f9-afd2-1ef573a4781c" (UID: "2a82b571-9098-44f9-afd2-1ef573a4781c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:20:43.550345 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:43.550317 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a82b571-9098-44f9-afd2-1ef573a4781c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2a82b571-9098-44f9-afd2-1ef573a4781c" (UID: "2a82b571-9098-44f9-afd2-1ef573a4781c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:20:43.550345 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:43.550330 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a82b571-9098-44f9-afd2-1ef573a4781c-kube-api-access-d47kr" (OuterVolumeSpecName: "kube-api-access-d47kr") pod "2a82b571-9098-44f9-afd2-1ef573a4781c" (UID: "2a82b571-9098-44f9-afd2-1ef573a4781c"). InnerVolumeSpecName "kube-api-access-d47kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:20:43.649988 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:43.649939 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a82b571-9098-44f9-afd2-1ef573a4781c-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:20:43.649988 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:43.649982 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d47kr\" (UniqueName: \"kubernetes.io/projected/2a82b571-9098-44f9-afd2-1ef573a4781c-kube-api-access-d47kr\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:20:43.649988 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:43.649995 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a82b571-9098-44f9-afd2-1ef573a4781c-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:20:44.093479 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:44.093436 2573 generic.go:356] "Generic (PLEG): container finished" podID="2a82b571-9098-44f9-afd2-1ef573a4781c" containerID="5e3761155962ed29d5db65fd9a78d49aa1a4185824ae34d5a3c4ea9764a371e3" exitCode=0 Jul 17 18:20:44.093479 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:44.093480 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" Jul 17 18:20:44.094019 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:44.093524 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" event={"ID":"2a82b571-9098-44f9-afd2-1ef573a4781c","Type":"ContainerDied","Data":"5e3761155962ed29d5db65fd9a78d49aa1a4185824ae34d5a3c4ea9764a371e3"} Jul 17 18:20:44.094019 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:44.093579 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs" event={"ID":"2a82b571-9098-44f9-afd2-1ef573a4781c","Type":"ContainerDied","Data":"0f0033cf9122d63f4cbe34c15ddb29dca61a7dff15c8eea5187190a9d42ae830"} Jul 17 18:20:44.094019 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:44.093604 2573 scope.go:117] "RemoveContainer" containerID="5e3761155962ed29d5db65fd9a78d49aa1a4185824ae34d5a3c4ea9764a371e3" Jul 17 18:20:44.102054 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:44.102038 2573 scope.go:117] "RemoveContainer" containerID="5e3761155962ed29d5db65fd9a78d49aa1a4185824ae34d5a3c4ea9764a371e3" Jul 17 18:20:44.102363 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:20:44.102343 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e3761155962ed29d5db65fd9a78d49aa1a4185824ae34d5a3c4ea9764a371e3\": container with ID starting with 5e3761155962ed29d5db65fd9a78d49aa1a4185824ae34d5a3c4ea9764a371e3 not found: ID does not exist" containerID="5e3761155962ed29d5db65fd9a78d49aa1a4185824ae34d5a3c4ea9764a371e3" Jul 17 18:20:44.102424 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:44.102384 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3761155962ed29d5db65fd9a78d49aa1a4185824ae34d5a3c4ea9764a371e3"} err="failed to get container status \"5e3761155962ed29d5db65fd9a78d49aa1a4185824ae34d5a3c4ea9764a371e3\": rpc error: code = NotFound desc = could not find container \"5e3761155962ed29d5db65fd9a78d49aa1a4185824ae34d5a3c4ea9764a371e3\": container with ID starting with 5e3761155962ed29d5db65fd9a78d49aa1a4185824ae34d5a3c4ea9764a371e3 not found: ID does not exist" Jul 17 18:20:44.114634 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:44.114592 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs"] Jul 17 18:20:44.119681 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:44.119661 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4699ff-9954bc95b-fntzs"] Jul 17 18:20:45.777303 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:45.777264 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a82b571-9098-44f9-afd2-1ef573a4781c" path="/var/lib/kubelet/pods/2a82b571-9098-44f9-afd2-1ef573a4781c/volumes" Jul 17 18:20:47.667868 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:47.667826 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:20:50.383284 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:50.383249 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:20:53.861457 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:53.861427 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:20:53.928990 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:53.928904 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:20:58.342992 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:58.342958 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s"] Jul 17 18:20:58.343455 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:58.343177 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" podUID="5897f210-3e09-484b-af11-042588e0a77d" containerName="istio-proxy" containerID="cri-o://85fd401dc5772bd9ef7c855726523b7b3725a27d32aa5bd588a8063d2e46972e" gracePeriod=30 Jul 17 18:20:58.368759 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:58.368729 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:20:58.589922 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:58.589891 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:20:59.291771 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:59.291733 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:20:59.391838 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:20:59.391803 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:21:03.590898 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.590869 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:21:03.644386 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.644304 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-workload-certs\") pod \"5897f210-3e09-484b-af11-042588e0a77d\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " Jul 17 18:21:03.644386 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.644348 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25qzd\" (UniqueName: \"kubernetes.io/projected/5897f210-3e09-484b-af11-042588e0a77d-kube-api-access-25qzd\") pod \"5897f210-3e09-484b-af11-042588e0a77d\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " Jul 17 18:21:03.644386 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.644380 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5897f210-3e09-484b-af11-042588e0a77d-istio-token\") pod \"5897f210-3e09-484b-af11-042588e0a77d\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " Jul 17 18:21:03.644688 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.644417 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-istio-data\") pod \"5897f210-3e09-484b-af11-042588e0a77d\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " Jul 17 18:21:03.644688 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.644579 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-istio-envoy\") pod \"5897f210-3e09-484b-af11-042588e0a77d\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " Jul 17 18:21:03.644688 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.644594 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5897f210-3e09-484b-af11-042588e0a77d" (UID: "5897f210-3e09-484b-af11-042588e0a77d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:21:03.644688 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.644650 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5897f210-3e09-484b-af11-042588e0a77d-istio-podinfo\") pod \"5897f210-3e09-484b-af11-042588e0a77d\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " Jul 17 18:21:03.644918 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.644723 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-workload-socket\") pod \"5897f210-3e09-484b-af11-042588e0a77d\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " Jul 17 18:21:03.644918 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.644778 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-credential-socket\") pod \"5897f210-3e09-484b-af11-042588e0a77d\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " Jul 17 18:21:03.644918 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.644830 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5897f210-3e09-484b-af11-042588e0a77d-istiod-ca-cert\") pod \"5897f210-3e09-484b-af11-042588e0a77d\" (UID: \"5897f210-3e09-484b-af11-042588e0a77d\") " Jul 17 18:21:03.645065 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.644981 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5897f210-3e09-484b-af11-042588e0a77d" (UID: "5897f210-3e09-484b-af11-042588e0a77d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:21:03.645260 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.645229 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:03.645260 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.645250 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5897f210-3e09-484b-af11-042588e0a77d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5897f210-3e09-484b-af11-042588e0a77d" (UID: "5897f210-3e09-484b-af11-042588e0a77d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:21:03.645382 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.645254 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:03.645438 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.645405 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5897f210-3e09-484b-af11-042588e0a77d" (UID: "5897f210-3e09-484b-af11-042588e0a77d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:21:03.645504 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.645445 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-istio-data" (OuterVolumeSpecName: "istio-data") pod "5897f210-3e09-484b-af11-042588e0a77d" (UID: "5897f210-3e09-484b-af11-042588e0a77d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:21:03.646771 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.646736 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5897f210-3e09-484b-af11-042588e0a77d-kube-api-access-25qzd" (OuterVolumeSpecName: "kube-api-access-25qzd") pod "5897f210-3e09-484b-af11-042588e0a77d" (UID: "5897f210-3e09-484b-af11-042588e0a77d"). InnerVolumeSpecName "kube-api-access-25qzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:21:03.646771 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.646763 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5897f210-3e09-484b-af11-042588e0a77d-istio-token" (OuterVolumeSpecName: "istio-token") pod "5897f210-3e09-484b-af11-042588e0a77d" (UID: "5897f210-3e09-484b-af11-042588e0a77d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:21:03.647041 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.647026 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5897f210-3e09-484b-af11-042588e0a77d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5897f210-3e09-484b-af11-042588e0a77d" (UID: "5897f210-3e09-484b-af11-042588e0a77d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 18:21:03.647041 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.647031 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5897f210-3e09-484b-af11-042588e0a77d" (UID: "5897f210-3e09-484b-af11-042588e0a77d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:21:03.746357 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.746313 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:03.746357 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.746349 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5897f210-3e09-484b-af11-042588e0a77d-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:03.746357 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.746363 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25qzd\" (UniqueName: \"kubernetes.io/projected/5897f210-3e09-484b-af11-042588e0a77d-kube-api-access-25qzd\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:03.746592 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.746375 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5897f210-3e09-484b-af11-042588e0a77d-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:03.746592 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.746387 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:03.746592 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.746398 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5897f210-3e09-484b-af11-042588e0a77d-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:03.746592 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:03.746408 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5897f210-3e09-484b-af11-042588e0a77d-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:04.193804 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:04.193769 2573 generic.go:356] "Generic (PLEG): container finished" podID="5897f210-3e09-484b-af11-042588e0a77d" containerID="85fd401dc5772bd9ef7c855726523b7b3725a27d32aa5bd588a8063d2e46972e" exitCode=0 Jul 17 18:21:04.194011 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:04.193834 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" Jul 17 18:21:04.194011 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:04.193848 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" event={"ID":"5897f210-3e09-484b-af11-042588e0a77d","Type":"ContainerDied","Data":"85fd401dc5772bd9ef7c855726523b7b3725a27d32aa5bd588a8063d2e46972e"} Jul 17 18:21:04.194011 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:04.193911 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s" event={"ID":"5897f210-3e09-484b-af11-042588e0a77d","Type":"ContainerDied","Data":"185a664a3329e20768c5b725bb7ab87a13fc64a805ed4d82c6d3c5e91223adce"} Jul 17 18:21:04.194011 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:04.193975 2573 scope.go:117] "RemoveContainer" containerID="85fd401dc5772bd9ef7c855726523b7b3725a27d32aa5bd588a8063d2e46972e" Jul 17 18:21:04.202573 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:04.202557 2573 scope.go:117] "RemoveContainer" containerID="30bf574212c6948b6ab24cbf9b57be56296b7b2dc62d8357e045410e97d668c1" Jul 17 18:21:04.210538 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:04.210476 2573 scope.go:117] "RemoveContainer" containerID="85fd401dc5772bd9ef7c855726523b7b3725a27d32aa5bd588a8063d2e46972e" Jul 17 18:21:04.210989 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:21:04.210959 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85fd401dc5772bd9ef7c855726523b7b3725a27d32aa5bd588a8063d2e46972e\": container with ID starting with 85fd401dc5772bd9ef7c855726523b7b3725a27d32aa5bd588a8063d2e46972e not found: ID does not exist" containerID="85fd401dc5772bd9ef7c855726523b7b3725a27d32aa5bd588a8063d2e46972e" Jul 17 18:21:04.211126 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:04.210997 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85fd401dc5772bd9ef7c855726523b7b3725a27d32aa5bd588a8063d2e46972e"} err="failed to get container status \"85fd401dc5772bd9ef7c855726523b7b3725a27d32aa5bd588a8063d2e46972e\": rpc error: code = NotFound desc = could not find container \"85fd401dc5772bd9ef7c855726523b7b3725a27d32aa5bd588a8063d2e46972e\": container with ID starting with 85fd401dc5772bd9ef7c855726523b7b3725a27d32aa5bd588a8063d2e46972e not found: ID does not exist" Jul 17 18:21:04.211126 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:04.211021 2573 scope.go:117] "RemoveContainer" containerID="30bf574212c6948b6ab24cbf9b57be56296b7b2dc62d8357e045410e97d668c1" Jul 17 18:21:04.211363 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:21:04.211336 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30bf574212c6948b6ab24cbf9b57be56296b7b2dc62d8357e045410e97d668c1\": container with ID starting with 30bf574212c6948b6ab24cbf9b57be56296b7b2dc62d8357e045410e97d668c1 not found: ID does not exist" containerID="30bf574212c6948b6ab24cbf9b57be56296b7b2dc62d8357e045410e97d668c1" Jul 17 18:21:04.211423 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:04.211373 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30bf574212c6948b6ab24cbf9b57be56296b7b2dc62d8357e045410e97d668c1"} err="failed to get container status \"30bf574212c6948b6ab24cbf9b57be56296b7b2dc62d8357e045410e97d668c1\": rpc error: code = NotFound desc = could not find container \"30bf574212c6948b6ab24cbf9b57be56296b7b2dc62d8357e045410e97d668c1\": container with ID starting with 30bf574212c6948b6ab24cbf9b57be56296b7b2dc62d8357e045410e97d668c1 not found: ID does not exist" Jul 17 18:21:04.213092 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:04.213075 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s"] Jul 17 18:21:04.217443 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:04.217412 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-4699ff-openshift-default-65f46c85c9-qjb5s"] Jul 17 18:21:05.777758 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:05.777727 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5897f210-3e09-484b-af11-042588e0a77d" path="/var/lib/kubelet/pods/5897f210-3e09-484b-af11-042588e0a77d/volumes" Jul 17 18:21:09.223230 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:09.223192 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz"] Jul 17 18:21:09.223735 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:09.223482 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" podUID="6863a941-0ecf-42b0-8179-52048edb547e" containerName="maas-api" containerID="cri-o://134b49d0982d235be081e21da690d2cf3d132db6a74b269d38f0caa557978156" gracePeriod=30 Jul 17 18:21:09.257707 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:09.257668 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:21:09.666003 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:09.665978 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" Jul 17 18:21:09.805721 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:09.805609 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tprhb\" (UniqueName: \"kubernetes.io/projected/6863a941-0ecf-42b0-8179-52048edb547e-kube-api-access-tprhb\") pod \"6863a941-0ecf-42b0-8179-52048edb547e\" (UID: \"6863a941-0ecf-42b0-8179-52048edb547e\") " Jul 17 18:21:09.805721 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:09.805682 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6863a941-0ecf-42b0-8179-52048edb547e-maas-api-tls\") pod \"6863a941-0ecf-42b0-8179-52048edb547e\" (UID: \"6863a941-0ecf-42b0-8179-52048edb547e\") " Jul 17 18:21:09.805908 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:09.805786 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6863a941-0ecf-42b0-8179-52048edb547e-openshift-service-ca\") pod \"6863a941-0ecf-42b0-8179-52048edb547e\" (UID: \"6863a941-0ecf-42b0-8179-52048edb547e\") " Jul 17 18:21:09.806196 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:09.806164 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6863a941-0ecf-42b0-8179-52048edb547e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6863a941-0ecf-42b0-8179-52048edb547e" (UID: "6863a941-0ecf-42b0-8179-52048edb547e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:21:09.807760 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:09.807731 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6863a941-0ecf-42b0-8179-52048edb547e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6863a941-0ecf-42b0-8179-52048edb547e" (UID: "6863a941-0ecf-42b0-8179-52048edb547e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:21:09.807851 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:09.807735 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6863a941-0ecf-42b0-8179-52048edb547e-kube-api-access-tprhb" (OuterVolumeSpecName: "kube-api-access-tprhb") pod "6863a941-0ecf-42b0-8179-52048edb547e" (UID: "6863a941-0ecf-42b0-8179-52048edb547e"). InnerVolumeSpecName "kube-api-access-tprhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:21:09.907758 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:09.907723 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tprhb\" (UniqueName: \"kubernetes.io/projected/6863a941-0ecf-42b0-8179-52048edb547e-kube-api-access-tprhb\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:09.907758 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:09.907752 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6863a941-0ecf-42b0-8179-52048edb547e-maas-api-tls\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:09.907758 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:09.907764 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6863a941-0ecf-42b0-8179-52048edb547e-openshift-service-ca\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:10.228530 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:10.228438 2573 generic.go:356] "Generic (PLEG): container finished" podID="6863a941-0ecf-42b0-8179-52048edb547e" containerID="134b49d0982d235be081e21da690d2cf3d132db6a74b269d38f0caa557978156" exitCode=0 Jul 17 18:21:10.228530 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:10.228487 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" event={"ID":"6863a941-0ecf-42b0-8179-52048edb547e","Type":"ContainerDied","Data":"134b49d0982d235be081e21da690d2cf3d132db6a74b269d38f0caa557978156"} Jul 17 18:21:10.229054 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:10.228545 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" event={"ID":"6863a941-0ecf-42b0-8179-52048edb547e","Type":"ContainerDied","Data":"b2423e15a79c67aed7c27be4ab714875e09a06dffe760c171849a554facd1b01"} Jul 17 18:21:10.229054 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:10.228501 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz" Jul 17 18:21:10.229054 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:10.228567 2573 scope.go:117] "RemoveContainer" containerID="134b49d0982d235be081e21da690d2cf3d132db6a74b269d38f0caa557978156" Jul 17 18:21:10.237806 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:10.237790 2573 scope.go:117] "RemoveContainer" containerID="134b49d0982d235be081e21da690d2cf3d132db6a74b269d38f0caa557978156" Jul 17 18:21:10.238005 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:21:10.237989 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134b49d0982d235be081e21da690d2cf3d132db6a74b269d38f0caa557978156\": container with ID starting with 134b49d0982d235be081e21da690d2cf3d132db6a74b269d38f0caa557978156 not found: ID does not exist" containerID="134b49d0982d235be081e21da690d2cf3d132db6a74b269d38f0caa557978156" Jul 17 18:21:10.238058 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:10.238013 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134b49d0982d235be081e21da690d2cf3d132db6a74b269d38f0caa557978156"} err="failed to get container status \"134b49d0982d235be081e21da690d2cf3d132db6a74b269d38f0caa557978156\": rpc error: code = NotFound desc = could not find container \"134b49d0982d235be081e21da690d2cf3d132db6a74b269d38f0caa557978156\": container with ID starting with 134b49d0982d235be081e21da690d2cf3d132db6a74b269d38f0caa557978156 not found: ID does not exist" Jul 17 18:21:10.250378 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:10.250342 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz"] Jul 17 18:21:10.254889 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:10.254859 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-22fb10-58fd948d54-pzxxz"] Jul 17 18:21:11.777937 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:11.777873 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6863a941-0ecf-42b0-8179-52048edb547e" path="/var/lib/kubelet/pods/6863a941-0ecf-42b0-8179-52048edb547e/volumes" Jul 17 18:21:13.816734 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:13.816682 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:21:14.300042 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:14.300009 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:21:14.400931 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:14.400896 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:21:19.777484 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:19.777453 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:21:19.878784 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:19.878751 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:21:24.489362 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:24.489325 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc"] Jul 17 18:21:24.489781 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:24.489551 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" podUID="206d67e3-7157-416a-a92f-22f661cdbe07" containerName="istio-proxy" containerID="cri-o://ba9178d55762072d6517c45988cac1fa5d824caacb2ab30c364abd32740e18e5" gracePeriod=30 Jul 17 18:21:24.509143 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:24.509111 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:21:24.732588 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:24.732546 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:21:25.143196 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:25.143168 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-dbfb57dbf-tjmnx_439e3fae-7a31-4cba-842a-5a961fc9039b/authorino/0.log" Jul 17 18:21:29.738788 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.738760 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:21:29.810577 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.810501 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmnd8\" (UniqueName: \"kubernetes.io/projected/206d67e3-7157-416a-a92f-22f661cdbe07-kube-api-access-rmnd8\") pod \"206d67e3-7157-416a-a92f-22f661cdbe07\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " Jul 17 18:21:29.810577 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.810563 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-workload-certs\") pod \"206d67e3-7157-416a-a92f-22f661cdbe07\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " Jul 17 18:21:29.810792 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.810664 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/206d67e3-7157-416a-a92f-22f661cdbe07-istio-token\") pod \"206d67e3-7157-416a-a92f-22f661cdbe07\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " Jul 17 18:21:29.810792 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.810714 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/206d67e3-7157-416a-a92f-22f661cdbe07-istio-podinfo\") pod \"206d67e3-7157-416a-a92f-22f661cdbe07\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " Jul 17 18:21:29.810792 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.810732 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/206d67e3-7157-416a-a92f-22f661cdbe07-istiod-ca-cert\") pod \"206d67e3-7157-416a-a92f-22f661cdbe07\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " Jul 17 18:21:29.810792 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.810751 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-credential-socket\") pod \"206d67e3-7157-416a-a92f-22f661cdbe07\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " Jul 17 18:21:29.811007 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.810798 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-istio-data\") pod \"206d67e3-7157-416a-a92f-22f661cdbe07\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " Jul 17 18:21:29.811007 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.810825 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-istio-envoy\") pod \"206d67e3-7157-416a-a92f-22f661cdbe07\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " Jul 17 18:21:29.811007 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.810853 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "206d67e3-7157-416a-a92f-22f661cdbe07" (UID: "206d67e3-7157-416a-a92f-22f661cdbe07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:21:29.811007 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.810867 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-workload-socket\") pod \"206d67e3-7157-416a-a92f-22f661cdbe07\" (UID: \"206d67e3-7157-416a-a92f-22f661cdbe07\") " Jul 17 18:21:29.811231 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.811053 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "206d67e3-7157-416a-a92f-22f661cdbe07" (UID: "206d67e3-7157-416a-a92f-22f661cdbe07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:21:29.811231 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.811093 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "206d67e3-7157-416a-a92f-22f661cdbe07" (UID: "206d67e3-7157-416a-a92f-22f661cdbe07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:21:29.811231 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.811119 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/206d67e3-7157-416a-a92f-22f661cdbe07-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "206d67e3-7157-416a-a92f-22f661cdbe07" (UID: "206d67e3-7157-416a-a92f-22f661cdbe07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:21:29.811389 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.811348 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-workload-certs\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:29.811389 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.811369 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/206d67e3-7157-416a-a92f-22f661cdbe07-istiod-ca-cert\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:29.811389 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.811381 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-credential-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:29.811546 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.811395 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-workload-socket\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:29.811769 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.811748 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-istio-data" (OuterVolumeSpecName: "istio-data") pod "206d67e3-7157-416a-a92f-22f661cdbe07" (UID: "206d67e3-7157-416a-a92f-22f661cdbe07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:21:29.812792 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.812769 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/206d67e3-7157-416a-a92f-22f661cdbe07-istio-token" (OuterVolumeSpecName: "istio-token") pod "206d67e3-7157-416a-a92f-22f661cdbe07" (UID: "206d67e3-7157-416a-a92f-22f661cdbe07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:21:29.812880 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.812843 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/206d67e3-7157-416a-a92f-22f661cdbe07-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "206d67e3-7157-416a-a92f-22f661cdbe07" (UID: "206d67e3-7157-416a-a92f-22f661cdbe07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 18:21:29.813421 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.813390 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "206d67e3-7157-416a-a92f-22f661cdbe07" (UID: "206d67e3-7157-416a-a92f-22f661cdbe07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:21:29.813421 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.813402 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/206d67e3-7157-416a-a92f-22f661cdbe07-kube-api-access-rmnd8" (OuterVolumeSpecName: "kube-api-access-rmnd8") pod "206d67e3-7157-416a-a92f-22f661cdbe07" (UID: "206d67e3-7157-416a-a92f-22f661cdbe07"). InnerVolumeSpecName "kube-api-access-rmnd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:21:29.912490 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.912464 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmnd8\" (UniqueName: \"kubernetes.io/projected/206d67e3-7157-416a-a92f-22f661cdbe07-kube-api-access-rmnd8\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:29.912490 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.912487 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/206d67e3-7157-416a-a92f-22f661cdbe07-istio-token\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:29.912490 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.912497 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/206d67e3-7157-416a-a92f-22f661cdbe07-istio-podinfo\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:29.912724 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.912504 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-istio-data\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:29.912724 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:29.912512 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/206d67e3-7157-416a-a92f-22f661cdbe07-istio-envoy\") on node \"ip-10-0-132-183.ec2.internal\" DevicePath \"\"" Jul 17 18:21:30.334328 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:30.334294 2573 generic.go:356] "Generic (PLEG): container finished" podID="206d67e3-7157-416a-a92f-22f661cdbe07" containerID="ba9178d55762072d6517c45988cac1fa5d824caacb2ab30c364abd32740e18e5" exitCode=0 Jul 17 18:21:30.334500 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:30.334340 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" Jul 17 18:21:30.334500 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:30.334359 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" event={"ID":"206d67e3-7157-416a-a92f-22f661cdbe07","Type":"ContainerDied","Data":"ba9178d55762072d6517c45988cac1fa5d824caacb2ab30c364abd32740e18e5"} Jul 17 18:21:30.334500 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:30.334401 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc" event={"ID":"206d67e3-7157-416a-a92f-22f661cdbe07","Type":"ContainerDied","Data":"c9dbd509ca3037ff7ba432601dc13b7f38bf5af29fd5fc4e66de8c0dc2764bbf"} Jul 17 18:21:30.334500 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:30.334423 2573 scope.go:117] "RemoveContainer" containerID="ba9178d55762072d6517c45988cac1fa5d824caacb2ab30c364abd32740e18e5" Jul 17 18:21:30.345353 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:30.345335 2573 scope.go:117] "RemoveContainer" containerID="04b6ad6c595a97d0440f17a402e34492861e4ceba9d6314f9d3bfa51b806ad28" Jul 17 18:21:30.352924 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:30.352905 2573 scope.go:117] "RemoveContainer" containerID="ba9178d55762072d6517c45988cac1fa5d824caacb2ab30c364abd32740e18e5" Jul 17 18:21:30.353141 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:21:30.353124 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9178d55762072d6517c45988cac1fa5d824caacb2ab30c364abd32740e18e5\": container with ID starting with ba9178d55762072d6517c45988cac1fa5d824caacb2ab30c364abd32740e18e5 not found: ID does not exist" containerID="ba9178d55762072d6517c45988cac1fa5d824caacb2ab30c364abd32740e18e5" Jul 17 18:21:30.353191 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:30.353150 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9178d55762072d6517c45988cac1fa5d824caacb2ab30c364abd32740e18e5"} err="failed to get container status \"ba9178d55762072d6517c45988cac1fa5d824caacb2ab30c364abd32740e18e5\": rpc error: code = NotFound desc = could not find container \"ba9178d55762072d6517c45988cac1fa5d824caacb2ab30c364abd32740e18e5\": container with ID starting with ba9178d55762072d6517c45988cac1fa5d824caacb2ab30c364abd32740e18e5 not found: ID does not exist" Jul 17 18:21:30.353191 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:30.353165 2573 scope.go:117] "RemoveContainer" containerID="04b6ad6c595a97d0440f17a402e34492861e4ceba9d6314f9d3bfa51b806ad28" Jul 17 18:21:30.353377 ip-10-0-132-183 kubenswrapper[2573]: E0717 18:21:30.353359 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b6ad6c595a97d0440f17a402e34492861e4ceba9d6314f9d3bfa51b806ad28\": container with ID starting with 04b6ad6c595a97d0440f17a402e34492861e4ceba9d6314f9d3bfa51b806ad28 not found: ID does not exist" containerID="04b6ad6c595a97d0440f17a402e34492861e4ceba9d6314f9d3bfa51b806ad28" Jul 17 18:21:30.353419 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:30.353381 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b6ad6c595a97d0440f17a402e34492861e4ceba9d6314f9d3bfa51b806ad28"} err="failed to get container status \"04b6ad6c595a97d0440f17a402e34492861e4ceba9d6314f9d3bfa51b806ad28\": rpc error: code = NotFound desc = could not find container \"04b6ad6c595a97d0440f17a402e34492861e4ceba9d6314f9d3bfa51b806ad28\": container with ID starting with 04b6ad6c595a97d0440f17a402e34492861e4ceba9d6314f9d3bfa51b806ad28 not found: ID does not exist" Jul 17 18:21:30.358350 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:30.358326 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc"] Jul 17 18:21:30.362316 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:30.362291 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-22fb10-openshift-default-865dfc98f9-vrkwc"] Jul 17 18:21:30.697122 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:30.697052 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-54f8d57c9d-hp6hv_76e8c837-dd34-4774-92c4-347adb3ca1a6/manager/0.log" Jul 17 18:21:31.030811 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:31.030780 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cbb795d5f-4qsx8_a2c1eaa4-5145-49ca-b9f0-d578671e5bc4/copy-manifests/0.log" Jul 17 18:21:31.037551 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:31.037529 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cbb795d5f-4qsx8_a2c1eaa4-5145-49ca-b9f0-d578671e5bc4/manager/0.log" Jul 17 18:21:31.638001 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:31.637973 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-c756574f6-ksprx_fa2cf22b-a4ca-4849-bf72-7982d45cca1b/maas-api/0.log" Jul 17 18:21:31.777220 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:31.777187 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="206d67e3-7157-416a-a92f-22f661cdbe07" path="/var/lib/kubelet/pods/206d67e3-7157-416a-a92f-22f661cdbe07/volumes" Jul 17 18:21:33.311576 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:33.311543 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-vltgs_3cda7224-c118-4808-8117-64c508e4f92a/postgres/0.log" Jul 17 18:21:34.591673 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:34.591610 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-dbfb57dbf-tjmnx_439e3fae-7a31-4cba-842a-5a961fc9039b/authorino/0.log" Jul 17 18:21:35.725463 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:35.725435 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf45zqbd_f9606969-27ab-45c3-938c-e661047077c8/istio-proxy/0.log" Jul 17 18:21:36.057800 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:36.057775 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d88846968-xdklf_e95b5d39-f981-4623-aa3f-762d3a24f3ca/kube-auth-proxy/0.log" Jul 17 18:21:36.175566 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:36.175537 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-x5wdk_00110376-3ab0-40d6-915f-92ac67509c1f/istio-proxy/1.log" Jul 17 18:21:42.992215 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992177 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5g29d/must-gather-7vjkv"] Jul 17 18:21:42.992613 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992599 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a82b571-9098-44f9-afd2-1ef573a4781c" containerName="maas-api" Jul 17 18:21:42.992613 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992615 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a82b571-9098-44f9-afd2-1ef573a4781c" containerName="maas-api" Jul 17 18:21:42.992613 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992637 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5897f210-3e09-484b-af11-042588e0a77d" containerName="istio-proxy" Jul 17 18:21:42.992762 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992643 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5897f210-3e09-484b-af11-042588e0a77d" containerName="istio-proxy" Jul 17 18:21:42.992762 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992659 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="206d67e3-7157-416a-a92f-22f661cdbe07" containerName="istio-proxy" Jul 17 18:21:42.992762 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992665 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="206d67e3-7157-416a-a92f-22f661cdbe07" containerName="istio-proxy" Jul 17 18:21:42.992762 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992672 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6863a941-0ecf-42b0-8179-52048edb547e" containerName="maas-api" Jul 17 18:21:42.992762 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992677 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6863a941-0ecf-42b0-8179-52048edb547e" containerName="maas-api" Jul 17 18:21:42.992762 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992684 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5897f210-3e09-484b-af11-042588e0a77d" containerName="istio-proxy" Jul 17 18:21:42.992762 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992689 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5897f210-3e09-484b-af11-042588e0a77d" containerName="istio-proxy" Jul 17 18:21:42.992762 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992702 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="206d67e3-7157-416a-a92f-22f661cdbe07" containerName="istio-proxy" Jul 17 18:21:42.992762 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992708 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="206d67e3-7157-416a-a92f-22f661cdbe07" containerName="istio-proxy" Jul 17 18:21:42.992762 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992720 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e584281-b5fa-46a5-af40-937908f6c380" containerName="test-curl-7573-140054093735232" Jul 17 18:21:42.992762 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992726 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e584281-b5fa-46a5-af40-937908f6c380" containerName="test-curl-7573-140054093735232" Jul 17 18:21:42.993133 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992796 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="206d67e3-7157-416a-a92f-22f661cdbe07" containerName="istio-proxy" Jul 17 18:21:42.993133 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992804 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6863a941-0ecf-42b0-8179-52048edb547e" containerName="maas-api" Jul 17 18:21:42.993133 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992812 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5897f210-3e09-484b-af11-042588e0a77d" containerName="istio-proxy" Jul 17 18:21:42.993133 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992818 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5897f210-3e09-484b-af11-042588e0a77d" containerName="istio-proxy" Jul 17 18:21:42.993133 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992826 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a82b571-9098-44f9-afd2-1ef573a4781c" containerName="maas-api" Jul 17 18:21:42.993133 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992834 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e584281-b5fa-46a5-af40-937908f6c380" containerName="test-curl-7573-140054093735232" Jul 17 18:21:42.993133 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.992847 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="206d67e3-7157-416a-a92f-22f661cdbe07" containerName="istio-proxy" Jul 17 18:21:42.998270 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:42.998249 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5g29d/must-gather-7vjkv" Jul 17 18:21:43.000749 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:43.000727 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5g29d\"/\"openshift-service-ca.crt\"" Jul 17 18:21:43.000867 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:43.000744 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5g29d\"/\"kube-root-ca.crt\"" Jul 17 18:21:43.000968 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:43.000956 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5g29d\"/\"default-dockercfg-rgpp8\"" Jul 17 18:21:43.005265 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:43.005233 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5g29d/must-gather-7vjkv"] Jul 17 18:21:43.006257 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:43.006228 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:21:43.108170 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:43.108138 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" limit=1024 Jul 17 18:21:43.134339 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:43.134305 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1851fa57-3dea-4f5e-81c7-35af1f516c61-must-gather-output\") pod \"must-gather-7vjkv\" (UID: \"1851fa57-3dea-4f5e-81c7-35af1f516c61\") " pod="openshift-must-gather-5g29d/must-gather-7vjkv" Jul 17 18:21:43.134488 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:43.134460 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwkz7\" (UniqueName: \"kubernetes.io/projected/1851fa57-3dea-4f5e-81c7-35af1f516c61-kube-api-access-wwkz7\") pod \"must-gather-7vjkv\" (UID: \"1851fa57-3dea-4f5e-81c7-35af1f516c61\") " pod="openshift-must-gather-5g29d/must-gather-7vjkv" Jul 17 18:21:43.235153 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:43.235111 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwkz7\" (UniqueName: \"kubernetes.io/projected/1851fa57-3dea-4f5e-81c7-35af1f516c61-kube-api-access-wwkz7\") pod \"must-gather-7vjkv\" (UID: \"1851fa57-3dea-4f5e-81c7-35af1f516c61\") " pod="openshift-must-gather-5g29d/must-gather-7vjkv" Jul 17 18:21:43.235359 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:43.235164 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1851fa57-3dea-4f5e-81c7-35af1f516c61-must-gather-output\") pod \"must-gather-7vjkv\" (UID: \"1851fa57-3dea-4f5e-81c7-35af1f516c61\") " pod="openshift-must-gather-5g29d/must-gather-7vjkv" Jul 17 18:21:43.235483 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:43.235465 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1851fa57-3dea-4f5e-81c7-35af1f516c61-must-gather-output\") pod \"must-gather-7vjkv\" (UID: \"1851fa57-3dea-4f5e-81c7-35af1f516c61\") " pod="openshift-must-gather-5g29d/must-gather-7vjkv" Jul 17 18:21:43.245165 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:43.245109 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwkz7\" (UniqueName: \"kubernetes.io/projected/1851fa57-3dea-4f5e-81c7-35af1f516c61-kube-api-access-wwkz7\") pod \"must-gather-7vjkv\" (UID: \"1851fa57-3dea-4f5e-81c7-35af1f516c61\") " pod="openshift-must-gather-5g29d/must-gather-7vjkv" Jul 17 18:21:43.309092 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:43.309056 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5g29d/must-gather-7vjkv" Jul 17 18:21:43.443539 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:43.443515 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5g29d/must-gather-7vjkv"] Jul 17 18:21:43.445548 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:21:43.445519 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1851fa57_3dea_4f5e_81c7_35af1f516c61.slice/crio-13327f1d21e237021beda3d04e00c809f056c1b494762dd2175cbc37317ad5db WatchSource:0}: Error finding container 13327f1d21e237021beda3d04e00c809f056c1b494762dd2175cbc37317ad5db: Status 404 returned error can't find the container with id 13327f1d21e237021beda3d04e00c809f056c1b494762dd2175cbc37317ad5db Jul 17 18:21:44.409512 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:44.409471 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5g29d/must-gather-7vjkv" event={"ID":"1851fa57-3dea-4f5e-81c7-35af1f516c61","Type":"ContainerStarted","Data":"9d334ceb863b434345ff1cf5726e0db10a0da9617cb15bb5e44131ebe4d41148"} Jul 17 18:21:44.410063 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:44.409534 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5g29d/must-gather-7vjkv" event={"ID":"1851fa57-3dea-4f5e-81c7-35af1f516c61","Type":"ContainerStarted","Data":"13327f1d21e237021beda3d04e00c809f056c1b494762dd2175cbc37317ad5db"} Jul 17 18:21:45.420562 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:45.420520 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5g29d/must-gather-7vjkv" event={"ID":"1851fa57-3dea-4f5e-81c7-35af1f516c61","Type":"ContainerStarted","Data":"482ae44d4a23487c7280866df18ee4cd0bafc4034d586802b491b4880a492d32"} Jul 17 18:21:45.436183 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:45.436133 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5g29d/must-gather-7vjkv" podStartSLOduration=2.612784607 podStartE2EDuration="3.436116133s" totalImagesPullingTime="823.331526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:21:45.435718468 +0000 UTC m=+3426.171142518" watchObservedRunningTime="2026-07-17 18:21:45.436116133 +0000 UTC m=+3426.171540230" Jul 17 18:21:45.832296 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:45.832259 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hr9d5_2a96d8c8-87eb-47c6-b427-0486a1b6eb23/global-pull-secret-syncer/0.log" Jul 17 18:21:45.967469 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:45.967431 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bk7mr_747c8133-308b-4cf0-99cd-d692c4a248ad/konnectivity-agent/0.log" Jul 17 18:21:46.054439 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:46.054411 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-183.ec2.internal_3e2d2ffede5ac360572d29fe6cd09b1b/haproxy/0.log" Jul 17 18:21:51.151350 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:51.151314 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-dbfb57dbf-tjmnx_439e3fae-7a31-4cba-842a-5a961fc9039b/authorino/0.log" Jul 17 18:21:52.956987 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:52.956959 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cccb9aa5-4d7e-460f-a66c-885c50c0fd61/alertmanager/0.log" Jul 17 18:21:52.988778 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:52.988687 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cccb9aa5-4d7e-460f-a66c-885c50c0fd61/config-reloader/0.log" Jul 17 18:21:53.013254 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.013227 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cccb9aa5-4d7e-460f-a66c-885c50c0fd61/kube-rbac-proxy-web/0.log" Jul 17 18:21:53.042319 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.042289 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cccb9aa5-4d7e-460f-a66c-885c50c0fd61/kube-rbac-proxy/0.log" Jul 17 18:21:53.073952 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.073905 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cccb9aa5-4d7e-460f-a66c-885c50c0fd61/kube-rbac-proxy-metric/0.log" Jul 17 18:21:53.101313 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.101280 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cccb9aa5-4d7e-460f-a66c-885c50c0fd61/prom-label-proxy/0.log" Jul 17 18:21:53.123899 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.123854 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cccb9aa5-4d7e-460f-a66c-885c50c0fd61/init-config-reloader/0.log" Jul 17 18:21:53.212061 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.211958 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-tb5v5_8a63170d-0b4c-466d-8171-8e08637042d5/kube-state-metrics/0.log" Jul 17 18:21:53.233146 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.233114 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-tb5v5_8a63170d-0b4c-466d-8171-8e08637042d5/kube-rbac-proxy-main/0.log" Jul 17 18:21:53.257216 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.257163 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-tb5v5_8a63170d-0b4c-466d-8171-8e08637042d5/kube-rbac-proxy-self/0.log" Jul 17 18:21:53.290197 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.290143 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-677bc7dc54-nvxff_e9120bc9-f36e-49b2-9477-9133a8c995b5/metrics-server/0.log" Jul 17 18:21:53.313635 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.313580 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-q5t65_5f12077a-e598-4040-a6f5-494f82567e3e/monitoring-plugin/0.log" Jul 17 18:21:53.550223 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.549772 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gt246_f6a2178f-b9d1-48d5-963d-49d79be84b3b/node-exporter/0.log" Jul 17 18:21:53.585765 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.585737 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gt246_f6a2178f-b9d1-48d5-963d-49d79be84b3b/kube-rbac-proxy/0.log" Jul 17 18:21:53.608883 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.608852 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gt246_f6a2178f-b9d1-48d5-963d-49d79be84b3b/init-textfile/0.log" Jul 17 18:21:53.653221 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.653190 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-wnqgx_04577b57-1d9c-49f0-bee5-55cbb5cf8d9b/kube-rbac-proxy-main/0.log" Jul 17 18:21:53.680952 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.680897 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-wnqgx_04577b57-1d9c-49f0-bee5-55cbb5cf8d9b/kube-rbac-proxy-self/0.log" Jul 17 18:21:53.701639 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.701597 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-wnqgx_04577b57-1d9c-49f0-bee5-55cbb5cf8d9b/openshift-state-metrics/0.log" Jul 17 18:21:53.738831 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.738788 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c987b6d-0ea4-48a1-85ae-1ece97c3de0b/prometheus/0.log" Jul 17 18:21:53.763493 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.763319 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c987b6d-0ea4-48a1-85ae-1ece97c3de0b/config-reloader/0.log" Jul 17 18:21:53.797458 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.797426 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c987b6d-0ea4-48a1-85ae-1ece97c3de0b/thanos-sidecar/0.log" Jul 17 18:21:53.826424 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.826389 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c987b6d-0ea4-48a1-85ae-1ece97c3de0b/kube-rbac-proxy-web/0.log" Jul 17 18:21:53.854827 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.854752 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c987b6d-0ea4-48a1-85ae-1ece97c3de0b/kube-rbac-proxy/0.log" Jul 17 18:21:53.889757 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.889727 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c987b6d-0ea4-48a1-85ae-1ece97c3de0b/kube-rbac-proxy-thanos/0.log" Jul 17 18:21:53.928862 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.928834 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c987b6d-0ea4-48a1-85ae-1ece97c3de0b/init-config-reloader/0.log" Jul 17 18:21:53.979944 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:53.979866 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-bbkrl_d2e961a8-8a7c-44fa-b5db-86fc6fd95bec/prometheus-operator/0.log" Jul 17 18:21:54.008641 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:54.006984 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-bbkrl_d2e961a8-8a7c-44fa-b5db-86fc6fd95bec/kube-rbac-proxy/0.log" Jul 17 18:21:54.047387 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:54.047280 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-lnkmx_c5a3ccba-9a63-4a3d-b7c5-9713e00881f5/prometheus-operator-admission-webhook/0.log" Jul 17 18:21:54.090564 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:54.090494 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fb58646b7-2srz5_bbd1bee5-1088-4975-af1e-9357c3b142ff/telemeter-client/0.log" Jul 17 18:21:54.136347 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:54.136285 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fb58646b7-2srz5_bbd1bee5-1088-4975-af1e-9357c3b142ff/reload/0.log" Jul 17 18:21:54.205131 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:54.205102 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fb58646b7-2srz5_bbd1bee5-1088-4975-af1e-9357c3b142ff/kube-rbac-proxy/0.log" Jul 17 18:21:54.299548 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:54.299461 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86cdc6ff4c-zdvsq_3963c290-c171-49e5-a957-a902e5ea7c46/thanos-query/0.log" Jul 17 18:21:54.359594 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:54.359560 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86cdc6ff4c-zdvsq_3963c290-c171-49e5-a957-a902e5ea7c46/kube-rbac-proxy-web/0.log" Jul 17 18:21:54.399489 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:54.399463 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86cdc6ff4c-zdvsq_3963c290-c171-49e5-a957-a902e5ea7c46/kube-rbac-proxy/0.log" Jul 17 18:21:54.437128 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:54.437099 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86cdc6ff4c-zdvsq_3963c290-c171-49e5-a957-a902e5ea7c46/prom-label-proxy/0.log" Jul 17 18:21:54.483576 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:54.483534 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86cdc6ff4c-zdvsq_3963c290-c171-49e5-a957-a902e5ea7c46/kube-rbac-proxy-rules/0.log" Jul 17 18:21:54.533003 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:54.532971 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86cdc6ff4c-zdvsq_3963c290-c171-49e5-a957-a902e5ea7c46/kube-rbac-proxy-metrics/0.log" Jul 17 18:21:56.238701 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.238669 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n"] Jul 17 18:21:56.246530 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.246497 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.254267 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.254238 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n"] Jul 17 18:21:56.309218 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.309174 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/377025c1-91ee-4876-8b82-3a08040efa55-podres\") pod \"perf-node-gather-daemonset-t7t8n\" (UID: \"377025c1-91ee-4876-8b82-3a08040efa55\") " pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.309413 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.309231 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/377025c1-91ee-4876-8b82-3a08040efa55-sys\") pod \"perf-node-gather-daemonset-t7t8n\" (UID: \"377025c1-91ee-4876-8b82-3a08040efa55\") " pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.309413 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.309312 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fccn\" (UniqueName: \"kubernetes.io/projected/377025c1-91ee-4876-8b82-3a08040efa55-kube-api-access-9fccn\") pod \"perf-node-gather-daemonset-t7t8n\" (UID: \"377025c1-91ee-4876-8b82-3a08040efa55\") " pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.309413 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.309398 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/377025c1-91ee-4876-8b82-3a08040efa55-proc\") pod \"perf-node-gather-daemonset-t7t8n\" (UID: \"377025c1-91ee-4876-8b82-3a08040efa55\") " pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.309598 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.309450 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/377025c1-91ee-4876-8b82-3a08040efa55-lib-modules\") pod \"perf-node-gather-daemonset-t7t8n\" (UID: \"377025c1-91ee-4876-8b82-3a08040efa55\") " pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.410666 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.410607 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/377025c1-91ee-4876-8b82-3a08040efa55-proc\") pod \"perf-node-gather-daemonset-t7t8n\" (UID: \"377025c1-91ee-4876-8b82-3a08040efa55\") " pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.410848 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.410698 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/377025c1-91ee-4876-8b82-3a08040efa55-lib-modules\") pod \"perf-node-gather-daemonset-t7t8n\" (UID: \"377025c1-91ee-4876-8b82-3a08040efa55\") " pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.410848 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.410714 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/377025c1-91ee-4876-8b82-3a08040efa55-proc\") pod \"perf-node-gather-daemonset-t7t8n\" (UID: \"377025c1-91ee-4876-8b82-3a08040efa55\") " pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.410848 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.410760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/377025c1-91ee-4876-8b82-3a08040efa55-podres\") pod \"perf-node-gather-daemonset-t7t8n\" (UID: \"377025c1-91ee-4876-8b82-3a08040efa55\") " pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.410848 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.410785 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/377025c1-91ee-4876-8b82-3a08040efa55-sys\") pod \"perf-node-gather-daemonset-t7t8n\" (UID: \"377025c1-91ee-4876-8b82-3a08040efa55\") " pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.411049 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.410849 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/377025c1-91ee-4876-8b82-3a08040efa55-lib-modules\") pod \"perf-node-gather-daemonset-t7t8n\" (UID: \"377025c1-91ee-4876-8b82-3a08040efa55\") " pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.411049 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.410854 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fccn\" (UniqueName: \"kubernetes.io/projected/377025c1-91ee-4876-8b82-3a08040efa55-kube-api-access-9fccn\") pod \"perf-node-gather-daemonset-t7t8n\" (UID: \"377025c1-91ee-4876-8b82-3a08040efa55\") " pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.411049 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.410898 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/377025c1-91ee-4876-8b82-3a08040efa55-sys\") pod \"perf-node-gather-daemonset-t7t8n\" (UID: \"377025c1-91ee-4876-8b82-3a08040efa55\") " pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.411049 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.410943 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/377025c1-91ee-4876-8b82-3a08040efa55-podres\") pod \"perf-node-gather-daemonset-t7t8n\" (UID: \"377025c1-91ee-4876-8b82-3a08040efa55\") " pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.420969 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.420908 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fccn\" (UniqueName: \"kubernetes.io/projected/377025c1-91ee-4876-8b82-3a08040efa55-kube-api-access-9fccn\") pod \"perf-node-gather-daemonset-t7t8n\" (UID: \"377025c1-91ee-4876-8b82-3a08040efa55\") " pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.565105 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.565003 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:56.748872 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:56.747890 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n"] Jul 17 18:21:56.751697 ip-10-0-132-183 kubenswrapper[2573]: W0717 18:21:56.749313 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod377025c1_91ee_4876_8b82_3a08040efa55.slice/crio-604d82193d361e0298f1d32ce7e2f4566aaaff419336661ab6ac3815609dcd37 WatchSource:0}: Error finding container 604d82193d361e0298f1d32ce7e2f4566aaaff419336661ab6ac3815609dcd37: Status 404 returned error can't find the container with id 604d82193d361e0298f1d32ce7e2f4566aaaff419336661ab6ac3815609dcd37 Jul 17 18:21:57.020902 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:57.020873 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5c57c47bcd-qsfqd_218a9863-42c8-4c5a-a019-225d6ade26fa/console/0.log" Jul 17 18:21:57.520925 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:57.520882 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" event={"ID":"377025c1-91ee-4876-8b82-3a08040efa55","Type":"ContainerStarted","Data":"08ab4b43d40e3b6531cf3b8171c431cae08fd8ae089bac806c4ba94c7e4942cd"} Jul 17 18:21:57.521411 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:57.520957 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" event={"ID":"377025c1-91ee-4876-8b82-3a08040efa55","Type":"ContainerStarted","Data":"604d82193d361e0298f1d32ce7e2f4566aaaff419336661ab6ac3815609dcd37"} Jul 17 18:21:57.521411 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:57.521191 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:21:57.540328 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:57.540274 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" podStartSLOduration=1.540259162 podStartE2EDuration="1.540259162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:21:57.537613703 +0000 UTC m=+3438.273037777" watchObservedRunningTime="2026-07-17 18:21:57.540259162 +0000 UTC m=+3438.275683242" Jul 17 18:21:58.818340 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:58.818308 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cf6c9_2fd76aa3-4339-43c6-a285-70b59ed5fd66/dns/0.log" Jul 17 18:21:58.838953 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:58.838927 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cf6c9_2fd76aa3-4339-43c6-a285-70b59ed5fd66/kube-rbac-proxy/0.log" Jul 17 18:21:58.863393 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:58.863365 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9bwpz_17bd4e15-f627-4ce8-8992-9e4e0a754e32/dns-node-resolver/0.log" Jul 17 18:21:59.500302 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:21:59.500270 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mmd7j_b905e39a-a951-4f8a-9479-348a1fea69e0/node-ca/0.log" Jul 17 18:22:00.445002 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:00.444970 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf45zqbd_f9606969-27ab-45c3-938c-e661047077c8/istio-proxy/0.log" Jul 17 18:22:00.623578 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:00.623550 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d88846968-xdklf_e95b5d39-f981-4623-aa3f-762d3a24f3ca/kube-auth-proxy/0.log" Jul 17 18:22:00.651914 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:00.651886 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-x5wdk_00110376-3ab0-40d6-915f-92ac67509c1f/istio-proxy/0.log" Jul 17 18:22:00.694578 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:00.694550 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-x5wdk_00110376-3ab0-40d6-915f-92ac67509c1f/istio-proxy/1.log" Jul 17 18:22:01.569875 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:01.569850 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sstpk_66a066f3-5b00-49a6-8b43-7498b829ca1e/serve-healthcheck-canary/0.log" Jul 17 18:22:02.155139 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:02.155098 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kpn4z_3e582980-4dc3-42ae-88f3-f9c09f0709c4/kube-rbac-proxy/0.log" Jul 17 18:22:02.188305 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:02.188279 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kpn4z_3e582980-4dc3-42ae-88f3-f9c09f0709c4/exporter/0.log" Jul 17 18:22:02.232879 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:02.232847 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kpn4z_3e582980-4dc3-42ae-88f3-f9c09f0709c4/extractor/0.log" Jul 17 18:22:03.537966 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:03.537934 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5g29d/perf-node-gather-daemonset-t7t8n" Jul 17 18:22:04.741606 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:04.741546 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-54f8d57c9d-hp6hv_76e8c837-dd34-4774-92c4-347adb3ca1a6/manager/0.log" Jul 17 18:22:04.864755 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:04.864719 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cbb795d5f-4qsx8_a2c1eaa4-5145-49ca-b9f0-d578671e5bc4/manager/0.log" Jul 17 18:22:04.890009 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:04.889981 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cbb795d5f-4qsx8_a2c1eaa4-5145-49ca-b9f0-d578671e5bc4/copy-manifests/0.log" Jul 17 18:22:06.215383 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:06.215293 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf45zqbd_f9606969-27ab-45c3-938c-e661047077c8/istio-proxy/0.log" Jul 17 18:22:06.360130 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:06.360104 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d88846968-xdklf_e95b5d39-f981-4623-aa3f-762d3a24f3ca/kube-auth-proxy/0.log" Jul 17 18:22:06.382115 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:06.382084 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-x5wdk_00110376-3ab0-40d6-915f-92ac67509c1f/istio-proxy/0.log" Jul 17 18:22:06.424020 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:06.423960 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-x5wdk_00110376-3ab0-40d6-915f-92ac67509c1f/istio-proxy/1.log" Jul 17 18:22:06.476134 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:06.476038 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6d77fc5c7f-t227k_6ed8fa43-d8c5-4d09-b772-a68f3bbe2804/manager/0.log" Jul 17 18:22:06.504927 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:06.504901 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jzvxh_25d12de3-1576-4a91-99e6-3b82842116f6/openshift-lws-operator/0.log" Jul 17 18:22:07.613081 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:07.613057 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf45zqbd_f9606969-27ab-45c3-938c-e661047077c8/istio-proxy/0.log" Jul 17 18:22:07.776263 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:07.776191 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d88846968-xdklf_e95b5d39-f981-4623-aa3f-762d3a24f3ca/kube-auth-proxy/0.log" Jul 17 18:22:07.797144 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:07.797114 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-x5wdk_00110376-3ab0-40d6-915f-92ac67509c1f/istio-proxy/0.log" Jul 17 18:22:07.843101 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:07.843065 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-x5wdk_00110376-3ab0-40d6-915f-92ac67509c1f/istio-proxy/1.log" Jul 17 18:22:13.592071 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:13.592031 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjj97_e92dd775-d49f-47b9-bf80-448ae55bb6f2/kube-multus-additional-cni-plugins/0.log" Jul 17 18:22:13.616080 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:13.616054 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjj97_e92dd775-d49f-47b9-bf80-448ae55bb6f2/egress-router-binary-copy/0.log" Jul 17 18:22:13.638559 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:13.638531 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjj97_e92dd775-d49f-47b9-bf80-448ae55bb6f2/cni-plugins/0.log" Jul 17 18:22:13.659727 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:13.659707 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjj97_e92dd775-d49f-47b9-bf80-448ae55bb6f2/bond-cni-plugin/0.log" Jul 17 18:22:13.679932 ip-10-0-132-183 kubenswrapper[2573]: I0717 18:22:13.679912 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjj97_e92dd775-d49f-47b9-bf80-448ae55bb6f2/routeoverride-cni/0.log"