Jul 16 21:09:41.766278 ip-10-0-137-133 systemd[1]: Starting Kubernetes Kubelet... Jul 16 21:09:42.191451 ip-10-0-137-133 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 21:09:42.191451 ip-10-0-137-133 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 21:09:42.191451 ip-10-0-137-133 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 21:09:42.191451 ip-10-0-137-133 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 21:09:42.191451 ip-10-0-137-133 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 21:09:42.191451 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.173332 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 21:09:42.191451 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179281 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:09:42.191451 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179296 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:09:42.191451 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179302 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:09:42.191451 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179306 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:09:42.191451 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179310 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:09:42.191451 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179315 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:09:42.191451 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179320 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:09:42.191451 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179332 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179339 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179345 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179349 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179354 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179358 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179363 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179367 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179371 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179375 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179378 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179382 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179386 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179390 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179394 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179398 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179403 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179407 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179411 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:09:42.192826 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179415 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179419 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179423 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179426 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179430 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179434 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179438 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179444 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179448 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179452 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179464 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179469 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179472 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179476 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179488 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179493 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179497 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179501 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179505 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:09:42.195445 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179509 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179513 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179517 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179520 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179524 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179528 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179532 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179540 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179544 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179548 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179553 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179557 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179561 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179565 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179569 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179576 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179580 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179585 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179589 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179594 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:09:42.196396 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179598 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179602 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179605 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179609 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179613 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179617 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179620 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179624 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179635 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179640 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179644 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179647 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179651 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179656 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179660 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179664 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179668 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179672 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179676 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179679 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:09:42.200316 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179683 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179687 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179691 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179695 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179700 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179704 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179707 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179711 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179715 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179719 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179724 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179728 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179731 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.179735 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180863 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180873 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180877 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180881 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180885 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:09:42.201261 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180889 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180893 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180904 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180909 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180913 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180917 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180920 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180924 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180928 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180932 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180936 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180940 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180944 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180948 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180971 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180977 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180982 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180987 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180992 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:09:42.290166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.180997 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:09:42.252832 ip-10-0-137-133 systemd[1]: Started Kubernetes Kubelet. Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181002 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181006 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181011 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181015 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181019 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181023 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181027 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181032 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181036 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181040 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181045 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181049 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181054 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181057 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181061 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181065 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181069 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181073 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181077 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:09:42.291014 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181082 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181085 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181089 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181093 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181098 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181102 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181106 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181109 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181113 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181118 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181123 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181127 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181133 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181139 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181143 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181147 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181152 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181156 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181160 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:09:42.295373 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181164 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181168 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181171 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181175 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181179 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181183 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181187 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181191 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181194 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181197 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181201 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181205 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181209 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181214 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181218 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181223 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181227 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181233 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181236 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181241 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:09:42.296422 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181245 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181248 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181252 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181256 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181260 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181264 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181268 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181272 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181276 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181279 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181283 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181287 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181290 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181294 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181298 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181302 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181857 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181869 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181873 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181876 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:09:42.297339 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181879 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181881 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181884 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181887 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181890 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181892 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181895 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181897 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181900 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181903 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181905 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181908 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181911 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181913 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181916 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181918 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181921 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181923 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181926 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181928 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:09:42.298200 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181931 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181933 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181938 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181941 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181946 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181950 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181967 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181970 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181973 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181977 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181980 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181982 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181985 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181987 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181990 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181992 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181996 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.181998 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182001 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:09:42.298922 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182003 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182006 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182008 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182011 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182013 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182016 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182019 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182021 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182023 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182026 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182028 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182031 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182034 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182036 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182038 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182041 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182043 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182046 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182049 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:09:42.299528 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182052 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182055 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182058 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182060 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182063 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182067 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182071 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182074 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182077 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182080 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182082 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182085 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182088 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182091 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182094 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182096 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182099 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182102 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182104 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:09:42.300084 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182107 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182109 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182112 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182114 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182117 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182119 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182122 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182124 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182127 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182129 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182132 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182135 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182137 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182140 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182143 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182145 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182148 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182150 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182227 2577 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182236 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182243 2577 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 21:09:44.320957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182247 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182251 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182254 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182258 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182263 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182266 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182269 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182273 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182278 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182282 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182287 2577 flags.go:64] FLAG: --cgroup-root="" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182291 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182296 2577 flags.go:64] FLAG: --client-ca-file="" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182299 2577 flags.go:64] FLAG: --cloud-provider="external" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182302 2577 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182306 2577 flags.go:64] FLAG: --cluster-domain="" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182309 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182312 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182315 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182318 2577 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182324 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182328 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182331 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 21:09:44.323323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182335 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182338 2577 flags.go:64] FLAG: --contention-profiling="false" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182341 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182344 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182347 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182349 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182354 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182358 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182360 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182363 2577 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182366 2577 flags.go:64] FLAG: --enable-server="true" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182369 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182374 2577 flags.go:64] FLAG: --event-burst="100" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182377 2577 flags.go:64] FLAG: --event-qps="50" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182380 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182383 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182386 2577 flags.go:64] FLAG: --eviction-hard="" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182389 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182392 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182395 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182398 2577 flags.go:64] FLAG: --eviction-soft="" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182400 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182403 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182406 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182409 2577 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 21:09:44.323971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182412 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182415 2577 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182418 2577 flags.go:64] FLAG: --feature-gates="" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182421 2577 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182426 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182429 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182432 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182435 2577 flags.go:64] FLAG: --healthz-port="10248" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182439 2577 flags.go:64] FLAG: --help="false" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182442 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182445 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182448 2577 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182450 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182454 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182457 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182461 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182463 2577 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182466 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182469 2577 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182472 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182475 2577 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182478 2577 flags.go:64] FLAG: --kube-reserved="" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182482 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182484 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 21:09:44.324687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182487 2577 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182490 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182493 2577 flags.go:64] FLAG: --lock-file="" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182496 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182499 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182502 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182507 2577 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182510 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182513 2577 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182516 2577 flags.go:64] FLAG: --logging-format="text" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182519 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182522 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182526 2577 flags.go:64] FLAG: --manifest-url="" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182532 2577 flags.go:64] FLAG: --manifest-url-header="" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182536 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182539 2577 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182542 2577 flags.go:64] FLAG: --max-pods="110" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182545 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182548 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182551 2577 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182554 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182557 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182560 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182563 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182570 2577 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 21:09:44.325327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182573 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182576 2577 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182579 2577 flags.go:64] FLAG: --pod-cidr="" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182582 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182587 2577 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182590 2577 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182593 2577 flags.go:64] FLAG: --pods-per-core="0" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182596 2577 flags.go:64] FLAG: --port="10250" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182599 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182602 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c307c69a02050302" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182605 2577 flags.go:64] FLAG: --qos-reserved="" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182608 2577 flags.go:64] FLAG: --read-only-port="10255" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182611 2577 flags.go:64] FLAG: --register-node="true" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182613 2577 flags.go:64] FLAG: --register-with-taints="" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182617 2577 flags.go:64] FLAG: --registry-burst="10" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182620 2577 flags.go:64] FLAG: --registry-qps="5" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182622 2577 flags.go:64] FLAG: --reserved-cpus="" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182625 2577 flags.go:64] FLAG: --reserved-memory="" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182629 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182633 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182636 2577 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182640 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182643 2577 flags.go:64] FLAG: --runonce="false" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182646 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182649 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 21:09:44.325962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182652 2577 flags.go:64] FLAG: --seccomp-default="false" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182655 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182658 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182660 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182663 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182666 2577 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182669 2577 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182673 2577 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182676 2577 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182678 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182681 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182684 2577 flags.go:64] FLAG: --system-cgroups="" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182700 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182706 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182709 2577 flags.go:64] FLAG: --tls-cert-file="" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182711 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182716 2577 flags.go:64] FLAG: --tls-min-version="" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182719 2577 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182721 2577 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182724 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182727 2577 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182730 2577 flags.go:64] FLAG: --v="2" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182734 2577 flags.go:64] FLAG: --version="false" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182737 2577 flags.go:64] FLAG: --vmodule="" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182741 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 21:09:44.326608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.182744 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182858 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182861 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182867 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182869 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182872 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182875 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182878 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182880 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182883 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182887 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182890 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182894 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182897 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182900 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182904 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182907 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182910 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182913 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:09:44.327217 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182916 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182920 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182922 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182925 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182928 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182930 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182933 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182935 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182938 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182940 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182943 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182945 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182948 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182950 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182974 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182978 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182982 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182984 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182987 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:09:44.327677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182989 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182992 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182995 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.182997 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183000 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183003 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183005 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183008 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183010 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183013 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183016 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183019 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183022 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183024 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183027 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183030 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183033 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183035 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183037 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:09:44.328139 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183040 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183042 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183045 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183048 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183050 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183054 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183057 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183060 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183064 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183067 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183072 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183075 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183078 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183082 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183085 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183088 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183090 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183093 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183095 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:09:44.328629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183098 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183100 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183103 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183105 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183108 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183111 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183113 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183116 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183118 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183121 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183123 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183126 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183128 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183131 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183133 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183136 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183139 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183141 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183144 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183146 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183149 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:09:44.329083 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183152 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:09:44.329629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183155 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:09:44.329629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.183159 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:09:44.329629 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.183846 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 21:09:44.329629 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.191730 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 21:09:44.329629 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.191748 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 21:09:44.329629 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.192778 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 21:09:44.329629 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.192790 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 21:09:44.329629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192845 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:09:44.329629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192850 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:09:44.329629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192855 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:09:44.329629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192858 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:09:44.329629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192861 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:09:44.329629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192863 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:09:44.329629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192866 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:09:44.329629 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192869 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192871 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192874 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192876 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192879 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192882 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192884 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192887 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192889 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192892 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192894 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192897 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192899 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192902 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192904 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192907 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192909 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192912 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192914 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192917 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:09:44.329999 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192921 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192923 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192926 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192929 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192931 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192934 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192936 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192939 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192941 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192944 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192946 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192949 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192966 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192971 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192974 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192976 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192979 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192981 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192984 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192996 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:09:44.330489 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.192999 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193001 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193006 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193009 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193013 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193016 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193019 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193022 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193025 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193028 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193031 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193034 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193037 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193040 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193043 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193046 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193050 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193054 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:09:44.331012 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193057 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193060 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193063 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193066 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193068 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193071 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193073 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193076 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193078 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193081 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193084 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193087 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193089 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193092 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193096 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193098 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193101 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193103 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193106 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193108 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:09:44.331464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193111 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:09:44.331934 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193113 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:09:44.331934 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193116 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:09:44.331934 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193118 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:09:44.331934 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193121 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:09:44.331934 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193124 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:09:44.331934 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193127 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:09:44.331934 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193129 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:09:44.331934 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193133 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:09:44.331934 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193137 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:09:44.331934 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193140 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:09:44.331934 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193143 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:09:44.331934 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193146 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:09:44.331934 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193149 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:09:44.331934 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.193154 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193261 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193265 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193269 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193272 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193274 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193277 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193280 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193283 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193286 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193289 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193291 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193294 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193298 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193300 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193303 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193306 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193308 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193311 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193314 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:09:44.332284 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193316 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193319 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193321 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193326 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193328 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193331 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193334 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193336 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193338 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193341 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193343 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193346 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193349 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193352 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193356 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193359 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193362 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193364 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193367 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:09:44.332763 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193370 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193373 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193375 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193378 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193381 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193383 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193385 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193389 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193391 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193394 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193397 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193399 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193401 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193404 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193406 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193410 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193415 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193418 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193420 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193423 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:09:44.333222 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193425 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193428 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193431 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193433 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193436 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193438 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193442 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193446 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193449 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193452 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193455 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193457 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193460 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193462 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193465 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193467 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193470 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193472 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193475 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:09:44.333718 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193477 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193480 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193483 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193485 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193487 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193490 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193492 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193495 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193501 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193504 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193507 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193509 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193512 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193514 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193517 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193519 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193522 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193524 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193527 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:09:44.334165 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193530 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:09:44.334735 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193532 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:09:44.334735 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:42.193534 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:09:44.334735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.193539 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 21:09:44.334735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.193630 2577 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 21:09:44.334735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.197832 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 21:09:44.334735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.199440 2577 server.go:1019] "Starting client certificate rotation" Jul 16 21:09:44.334735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.199533 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 21:09:44.334735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.199574 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 21:09:44.334735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.204283 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 21:09:44.334735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.205352 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 21:09:44.334735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.218944 2577 log.go:25] "Validated CRI v1 runtime API" Jul 16 21:09:44.334735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.221226 2577 log.go:25] "Validated CRI v1 image API" Jul 16 21:09:44.334735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.222058 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 21:09:44.335162 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.224523 2577 fs.go:135] Filesystem UUIDs: map[19c29017-069e-4e00-a9e6-410647865db2:/dev/nvme0n1p4 43563cab-47d5-45ca-acf9-500213c2f651:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 16 21:09:44.335162 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.224542 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.230841 2577 manager.go:216] Machine: {Timestamp:2026-07-16 21:09:42.229577444 +0000 UTC m=+0.348913402 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102137 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c5e7ec10ae18f87f2e8dcb506d157 SystemUUID:ec2c5e7e-c10a-e18f-87f2-e8dcb506d157 BootID:ea43aaae-98df-4c0c-9fb8-04bcfd10b2a3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7e:04:f5:b7:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7e:04:f5:b7:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:b4:04:e9:a7:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.230974 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.231051 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.234052 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.234079 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-133.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.234238 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.235380 2577 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.235393 2577 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.235400 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.236403 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.237738 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.240820 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.240947 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.240994 2577 kubelet.go:486] "Attempting to sync node with API server" Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.241007 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.241020 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 21:09:44.335271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.241028 2577 kubelet.go:398] "Adding apiserver pod source" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.241041 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.241698 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.241716 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.245798 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.247691 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.247727 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.248384 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.248403 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.248411 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.248419 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.248430 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.248438 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.248449 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.248456 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.248465 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.248472 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.248483 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.248491 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.248518 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.248525 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.252191 2577 server.go:1268] "Started kubelet" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.255490 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.255528 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:42.255661 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:42.255741 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-133.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.255876 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.252189 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-133.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.257802 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.258095 2577 server.go:310] "Adding debug handlers to kubelet server" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:42.257782 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-133.ec2.internal.18c2e15abe8a22a2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-133.ec2.internal,UID:ip-10-0-137-133.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-133.ec2.internal,},FirstTimestamp:2026-07-16 21:09:42.252167842 +0000 UTC m=+0.371503781,LastTimestamp:2026-07-16 21:09:42.252167842 +0000 UTC m=+0.371503781,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-133.ec2.internal,}" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:42.261525 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.262743 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.263018 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:42.265253 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-133.ec2.internal\" not found" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.265807 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.265824 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.266040 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.266115 2577 reconstruct.go:97] "Volume reconstruction finished" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.266122 2577 reconciler.go:29] "Reconciler: start to sync state" Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.266833 2577 factory.go:55] Registering systemd factory Jul 16 21:09:44.335829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.266845 2577 factory.go:223] Registration of the systemd container factory successfully Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.267021 2577 factory.go:153] Registering CRI-O factory Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.267032 2577 factory.go:223] Registration of the crio container factory successfully Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.267087 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.267104 2577 factory.go:103] Registering Raw factory Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.267114 2577 manager.go:1196] Started watching for new ooms in manager Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.267411 2577 manager.go:349] Starting recovery of all containers Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:42.270212 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-133.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:42.270367 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.280500 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cktpx" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.280540 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cktpx" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.295912 2577 manager.go:354] Recovery completed Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.301077 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.302936 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.302993 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.303008 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.304167 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.304179 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.304197 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.305439 2577 policy_none.go:49] "None policy: Start" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.305456 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.305469 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.306506 2577 policy_none.go:47] "Start" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.307133 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.307751 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.307770 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:42.307780 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:44.310339 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-133.ec2.internal\" not found" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.334803 2577 manager.go:337] "Starting Device Plugin manager" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:44.334958 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.334968 2577 server.go:87] "Starting device plugin registration server" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.335170 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.335196 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.335308 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.335440 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.335457 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:44.336462 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 21:09:44.336912 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:44.336511 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-133.ec2.internal\" not found" Jul 16 21:09:44.422959 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:44.422897 2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-133.ec2.internal\" not found" node="ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.436018 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.435992 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:09:44.436837 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.436821 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:09:44.436911 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.436851 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:09:44.436911 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.436862 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:09:44.436911 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.436886 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.443131 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.443111 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 21:09:44.444218 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.444205 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 21:09:44.444280 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.444240 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 21:09:44.444280 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.444257 2577 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 21:09:44.444346 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:44.444288 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 21:09:44.445752 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.445730 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.445837 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:44.445760 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-133.ec2.internal\": node \"ip-10-0-137-133.ec2.internal\" not found" Jul 16 21:09:44.450404 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.450382 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:09:44.473488 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:44.473464 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-133.ec2.internal\" not found" Jul 16 21:09:44.544583 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.544540 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-133.ec2.internal"] Jul 16 21:09:44.544705 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.544644 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:09:44.545573 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.545555 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:09:44.545639 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.545586 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:09:44.545639 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.545599 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:09:44.546896 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.546883 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:09:44.547018 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.547004 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.547063 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.547034 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:09:44.547585 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.547565 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:09:44.547669 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.547590 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:09:44.547669 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.547599 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:09:44.547669 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.547565 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:09:44.547759 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.547669 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:09:44.547759 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.547681 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:09:44.548811 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.548798 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.548859 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.548823 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:09:44.549499 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.549484 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:09:44.549566 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.549511 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:09:44.549566 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.549521 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:09:44.574119 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:44.574092 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-133.ec2.internal\" not found" Jul 16 21:09:44.575889 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:44.575871 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-133.ec2.internal\" not found" node="ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.580178 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:44.580162 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-133.ec2.internal\" not found" node="ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.613633 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.613607 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86c218ba58dac56ce79b99c5b33e3622-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal\" (UID: \"86c218ba58dac56ce79b99c5b33e3622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.613720 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.613638 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86c218ba58dac56ce79b99c5b33e3622-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal\" (UID: \"86c218ba58dac56ce79b99c5b33e3622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.613720 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.613660 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd1e9b97c4c8ceff5fa6e561d0396716-config\") pod \"kube-apiserver-proxy-ip-10-0-137-133.ec2.internal\" (UID: \"cd1e9b97c4c8ceff5fa6e561d0396716\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.674733 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:44.674678 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-133.ec2.internal\" not found" Jul 16 21:09:44.714592 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.714568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86c218ba58dac56ce79b99c5b33e3622-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal\" (UID: \"86c218ba58dac56ce79b99c5b33e3622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.714698 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.714593 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86c218ba58dac56ce79b99c5b33e3622-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal\" (UID: \"86c218ba58dac56ce79b99c5b33e3622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.714698 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.714611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd1e9b97c4c8ceff5fa6e561d0396716-config\") pod \"kube-apiserver-proxy-ip-10-0-137-133.ec2.internal\" (UID: \"cd1e9b97c4c8ceff5fa6e561d0396716\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.714698 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.714667 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cd1e9b97c4c8ceff5fa6e561d0396716-config\") pod \"kube-apiserver-proxy-ip-10-0-137-133.ec2.internal\" (UID: \"cd1e9b97c4c8ceff5fa6e561d0396716\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.714698 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.714680 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86c218ba58dac56ce79b99c5b33e3622-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal\" (UID: \"86c218ba58dac56ce79b99c5b33e3622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.714816 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.714685 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86c218ba58dac56ce79b99c5b33e3622-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal\" (UID: \"86c218ba58dac56ce79b99c5b33e3622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.775738 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:44.775703 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-133.ec2.internal\" not found" Jul 16 21:09:44.876209 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:44.876156 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-133.ec2.internal\" not found" Jul 16 21:09:44.878358 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.878319 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.884416 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:44.884386 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-133.ec2.internal" Jul 16 21:09:44.976774 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:44.976694 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-133.ec2.internal\" not found" Jul 16 21:09:45.077212 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:45.077166 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-133.ec2.internal\" not found" Jul 16 21:09:45.142592 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:45.142573 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 21:09:45.143010 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:45.142706 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 21:09:45.177891 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:45.177869 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-133.ec2.internal\" not found" Jul 16 21:09:45.203411 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:45.203387 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:09:45.229733 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:45.229660 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 21:04:44 +0000 UTC" deadline="2028-01-28 02:12:56.971349073 +0000 UTC" Jul 16 21:09:45.229733 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:45.229685 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13445h3m11.741666363s" Jul 16 21:09:45.278858 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:45.278831 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-133.ec2.internal\" not found" Jul 16 21:09:45.379344 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:45.379312 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-133.ec2.internal\" not found" Jul 16 21:09:45.450861 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:45.450826 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c218ba58dac56ce79b99c5b33e3622.slice/crio-3272cd96fafac8847012d22f466eca602a110d391548c0f3fcd3b34f58a63a61 WatchSource:0}: Error finding container 3272cd96fafac8847012d22f466eca602a110d391548c0f3fcd3b34f58a63a61: Status 404 returned error can't find the container with id 3272cd96fafac8847012d22f466eca602a110d391548c0f3fcd3b34f58a63a61 Jul 16 21:09:45.451322 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:45.451302 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1e9b97c4c8ceff5fa6e561d0396716.slice/crio-b8995ab056102a597e6c7c8ab9b487c620d4c69ce6c81af18e9eaa8df301665c WatchSource:0}: Error finding container b8995ab056102a597e6c7c8ab9b487c620d4c69ce6c81af18e9eaa8df301665c: Status 404 returned error can't find the container with id b8995ab056102a597e6c7c8ab9b487c620d4c69ce6c81af18e9eaa8df301665c Jul 16 21:09:45.454549 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:45.454528 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:09:45.455822 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:45.455808 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:09:45.511656 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:45.511601 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-133.ec2.internal" Jul 16 21:09:45.524298 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:45.524283 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 21:09:45.526030 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:45.526018 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal" Jul 16 21:09:45.542178 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:45.542164 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 21:09:45.571372 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:45.571353 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:09:46.187027 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.186996 2577 apiserver.go:52] "Watching apiserver" Jul 16 21:09:46.195430 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.195406 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 21:09:46.197553 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.197515 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-133.ec2.internal","openshift-dns/node-resolver-bb7qw","openshift-image-registry/node-ca-bcb4c","openshift-multus/multus-smplh","openshift-multus/network-metrics-daemon-twm75","openshift-network-operator/iptables-alerter-k44hd","kube-system/konnectivity-agent-s2pdw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd","openshift-cluster-node-tuning-operator/tuned-2nd6g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal","openshift-multus/multus-additional-cni-plugins-gm6dn","openshift-network-diagnostics/network-check-target-dbknf","openshift-ovn-kubernetes/ovnkube-node-d5lc2","kube-system/global-pull-secret-syncer-2xbp6"] Jul 16 21:09:46.199039 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.199015 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s2pdw" Jul 16 21:09:46.200864 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.200845 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.202093 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.202072 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bcb4c" Jul 16 21:09:46.203295 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.203279 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-smplh" Jul 16 21:09:46.204026 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.204006 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 21:09:46.204115 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.204065 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kfslq\"" Jul 16 21:09:46.204115 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.204066 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 21:09:46.204616 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.204599 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:09:46.204696 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:46.204668 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:09:46.205915 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.205897 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k44hd" Jul 16 21:09:46.206557 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.206543 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 21:09:46.207798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.207482 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 21:09:46.207798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.207545 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 21:09:46.207798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.207629 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pfzrv\"" Jul 16 21:09:46.207798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.207549 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 21:09:46.208006 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.207921 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.208006 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.207951 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 21:09:46.208102 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.208061 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rkl7z\"" Jul 16 21:09:46.208811 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.208784 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 21:09:46.208811 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.208792 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 21:09:46.208811 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.208812 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 21:09:46.209036 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.208824 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 21:09:46.209036 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.208836 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 21:09:46.209036 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.208872 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 21:09:46.209036 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.208838 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hswhk\"" Jul 16 21:09:46.209261 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.209168 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:46.209261 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:46.209234 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:09:46.209635 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.209618 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 21:09:46.209719 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.209702 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 21:09:46.209798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.209737 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 21:09:46.209798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.209769 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:09:46.209798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.209800 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 21:09:46.210029 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.210015 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jkc6t\"" Jul 16 21:09:46.210532 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.210513 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bb7qw" Jul 16 21:09:46.212463 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.212446 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.214061 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.214045 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.214557 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.214481 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2lpnb\"" Jul 16 21:09:46.214827 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.214806 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 21:09:46.215397 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.215330 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 21:09:46.215607 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.215415 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 21:09:46.217812 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.215842 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:09:46.217812 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.216308 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rz57f\"" Jul 16 21:09:46.217812 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.216331 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4b7t8\"" Jul 16 21:09:46.217812 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.216491 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 21:09:46.217812 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.216557 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 21:09:46.217812 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.217103 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 21:09:46.217812 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.217158 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 21:09:46.217812 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.217440 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:09:46.217812 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.217638 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v8bvx\"" Jul 16 21:09:46.217812 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:46.217667 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:09:46.217812 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.217778 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 21:09:46.218359 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.217990 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 21:09:46.221029 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221006 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.221134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221043 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d14ceb5-180d-4cd4-9653-70954a4f935b-serviceca\") pod \"node-ca-bcb4c\" (UID: \"1d14ceb5-180d-4cd4-9653-70954a4f935b\") " pod="openshift-image-registry/node-ca-bcb4c" Jul 16 21:09:46.221134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221069 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-multus-conf-dir\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.221134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221094 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmkcr\" (UniqueName: \"kubernetes.io/projected/1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3-kube-api-access-lmkcr\") pod \"iptables-alerter-k44hd\" (UID: \"1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3\") " pod="openshift-network-operator/iptables-alerter-k44hd" Jul 16 21:09:46.221134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221118 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65dl8\" (UniqueName: \"kubernetes.io/projected/9a4e202f-99dc-4068-9496-acb28f0552b5-kube-api-access-65dl8\") pod \"network-metrics-daemon-twm75\" (UID: \"9a4e202f-99dc-4068-9496-acb28f0552b5\") " pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:09:46.221325 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1ae55087-4ec5-476b-a36e-a46fc3ae4479-ovnkube-config\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.221325 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221213 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05328ec6-9d7f-41fa-b8a2-733ff3025de8-cni-binary-copy\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.221325 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221252 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret\") pod \"global-pull-secret-syncer-2xbp6\" (UID: \"3bd19c30-8019-4896-aa14-62214c3e042c\") " pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:46.221325 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221300 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-log-socket\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.221487 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221340 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-var-lib-kubelet\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.221487 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-systemd\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.221487 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221395 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-sys\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.221487 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221450 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-tmp\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.221487 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-run-openvswitch\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.221700 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-run-k8s-cni-cncf-io\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.221700 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221526 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.221700 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221544 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-registration-dir\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.221700 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221559 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-device-dir\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.221700 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-host\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.221700 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221610 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-var-lib-openvswitch\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.221700 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-run-ovn\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.221700 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221670 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1ae55087-4ec5-476b-a36e-a46fc3ae4479-env-overrides\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.221987 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221713 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-cnibin\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.221987 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3bd19c30-8019-4896-aa14-62214c3e042c-kubelet-config\") pod \"global-pull-secret-syncer-2xbp6\" (UID: \"3bd19c30-8019-4896-aa14-62214c3e042c\") " pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:46.221987 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221759 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs\") pod \"network-metrics-daemon-twm75\" (UID: \"9a4e202f-99dc-4068-9496-acb28f0552b5\") " pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:09:46.221987 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221799 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21f02b78-840d-4d87-a3da-ef3a063424a2-konnectivity-ca\") pod \"konnectivity-agent-s2pdw\" (UID: \"21f02b78-840d-4d87-a3da-ef3a063424a2\") " pod="kube-system/konnectivity-agent-s2pdw" Jul 16 21:09:46.221987 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221843 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-run-netns\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.221987 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221871 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-run-systemd\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.221987 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-cni-bin\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.221987 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221917 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glcm4\" (UniqueName: \"kubernetes.io/projected/1ae55087-4ec5-476b-a36e-a46fc3ae4479-kube-api-access-glcm4\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.221987 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221954 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-sysctl-conf\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.221987 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.221980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-slash\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.222355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-etc-kubernetes\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.222355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222029 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3-iptables-alerter-script\") pod \"iptables-alerter-k44hd\" (UID: \"1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3\") " pod="openshift-network-operator/iptables-alerter-k44hd" Jul 16 21:09:46.222355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222051 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-sysconfig\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.222355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222089 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-var-lib-kubelet\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.222355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222114 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eda93f33-be42-4a44-ab7c-1c1e93d65b74-tmp-dir\") pod \"node-resolver-bb7qw\" (UID: \"eda93f33-be42-4a44-ab7c-1c1e93d65b74\") " pod="openshift-dns/node-resolver-bb7qw" Jul 16 21:09:46.222355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222136 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-etc-openvswitch\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.222355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222151 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-system-cni-dir\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.222355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3bd19c30-8019-4896-aa14-62214c3e042c-dbus\") pod \"global-pull-secret-syncer-2xbp6\" (UID: \"3bd19c30-8019-4896-aa14-62214c3e042c\") " pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:46.222355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222223 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58stf\" (UniqueName: \"kubernetes.io/projected/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-kube-api-access-58stf\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.222355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-run-ovn-kubernetes\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.222355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-run-netns\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.222355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222303 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxshd\" (UniqueName: \"kubernetes.io/projected/eda93f33-be42-4a44-ab7c-1c1e93d65b74-kube-api-access-rxshd\") pod \"node-resolver-bb7qw\" (UID: \"eda93f33-be42-4a44-ab7c-1c1e93d65b74\") " pod="openshift-dns/node-resolver-bb7qw" Jul 16 21:09:46.222355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-node-log\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.222355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1ae55087-4ec5-476b-a36e-a46fc3ae4479-ovn-node-metrics-cert\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.222355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222358 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1ae55087-4ec5-476b-a36e-a46fc3ae4479-ovnkube-script-lib\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222379 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-kubernetes\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-sysctl-d\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222419 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eda93f33-be42-4a44-ab7c-1c1e93d65b74-hosts-file\") pod \"node-resolver-bb7qw\" (UID: \"eda93f33-be42-4a44-ab7c-1c1e93d65b74\") " pod="openshift-dns/node-resolver-bb7qw" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222440 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-kubelet\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvbdv\" (UniqueName: \"kubernetes.io/projected/05328ec6-9d7f-41fa-b8a2-733ff3025de8-kube-api-access-hvbdv\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222485 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3-host-slash\") pod \"iptables-alerter-k44hd\" (UID: \"1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3\") " pod="openshift-network-operator/iptables-alerter-k44hd" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222506 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-cni-netd\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222536 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr87w\" (UniqueName: \"kubernetes.io/projected/1d14ceb5-180d-4cd4-9653-70954a4f935b-kube-api-access-tr87w\") pod \"node-ca-bcb4c\" (UID: \"1d14ceb5-180d-4cd4-9653-70954a4f935b\") " pod="openshift-image-registry/node-ca-bcb4c" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222557 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-var-lib-cni-multus\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222580 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05328ec6-9d7f-41fa-b8a2-733ff3025de8-multus-daemon-config\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222620 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-sys-fs\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222663 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lsf8\" (UniqueName: \"kubernetes.io/projected/3a13edcb-9f53-409a-860f-0d1275da1cc4-kube-api-access-7lsf8\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-run\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-tuned\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222769 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-multus-cni-dir\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-os-release\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.222895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-multus-socket-dir-parent\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.223523 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222832 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-run-multus-certs\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.223523 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222851 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21f02b78-840d-4d87-a3da-ef3a063424a2-agent-certs\") pod \"konnectivity-agent-s2pdw\" (UID: \"21f02b78-840d-4d87-a3da-ef3a063424a2\") " pod="kube-system/konnectivity-agent-s2pdw" Jul 16 21:09:46.223523 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d14ceb5-180d-4cd4-9653-70954a4f935b-host\") pod \"node-ca-bcb4c\" (UID: \"1d14ceb5-180d-4cd4-9653-70954a4f935b\") " pod="openshift-image-registry/node-ca-bcb4c" Jul 16 21:09:46.223523 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-modprobe-d\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.223523 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222890 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-lib-modules\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.223523 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222914 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-var-lib-cni-bin\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.223523 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222949 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-hostroot\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.223523 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222972 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-socket-dir\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.223523 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.222987 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.223523 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.223006 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-systemd-units\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.228097 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.228074 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 21:09:46.263019 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.262859 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t4stl" Jul 16 21:09:46.277755 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.277732 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t4stl" Jul 16 21:09:46.311825 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.311801 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 21:09:46.323333 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-node-log\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.323482 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1ae55087-4ec5-476b-a36e-a46fc3ae4479-ovn-node-metrics-cert\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.323482 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323417 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-node-log\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.323598 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1ae55087-4ec5-476b-a36e-a46fc3ae4479-ovnkube-script-lib\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.323598 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-kubernetes\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.323598 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-sysctl-d\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.323598 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323558 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eda93f33-be42-4a44-ab7c-1c1e93d65b74-hosts-file\") pod \"node-resolver-bb7qw\" (UID: \"eda93f33-be42-4a44-ab7c-1c1e93d65b74\") " pod="openshift-dns/node-resolver-bb7qw" Jul 16 21:09:46.323598 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323586 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z888n\" (UniqueName: \"kubernetes.io/projected/244e9d5d-772a-4da3-8552-8e1c42165724-kube-api-access-z888n\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.323843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-kubelet\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.323843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvbdv\" (UniqueName: \"kubernetes.io/projected/05328ec6-9d7f-41fa-b8a2-733ff3025de8-kube-api-access-hvbdv\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.323843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323672 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3-host-slash\") pod \"iptables-alerter-k44hd\" (UID: \"1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3\") " pod="openshift-network-operator/iptables-alerter-k44hd" Jul 16 21:09:46.323843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-cni-netd\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.323843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr87w\" (UniqueName: \"kubernetes.io/projected/1d14ceb5-180d-4cd4-9653-70954a4f935b-kube-api-access-tr87w\") pod \"node-ca-bcb4c\" (UID: \"1d14ceb5-180d-4cd4-9653-70954a4f935b\") " pod="openshift-image-registry/node-ca-bcb4c" Jul 16 21:09:46.323843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323751 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 21:09:46.323843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323805 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-var-lib-cni-multus\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.324174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323899 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-kubelet\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.324174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323992 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-sysctl-d\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.324174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324046 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-cni-netd\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.324174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324096 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-kubernetes\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.324174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324099 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eda93f33-be42-4a44-ab7c-1c1e93d65b74-hosts-file\") pod \"node-resolver-bb7qw\" (UID: \"eda93f33-be42-4a44-ab7c-1c1e93d65b74\") " pod="openshift-dns/node-resolver-bb7qw" Jul 16 21:09:46.324174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324137 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3-host-slash\") pod \"iptables-alerter-k44hd\" (UID: \"1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3\") " pod="openshift-network-operator/iptables-alerter-k44hd" Jul 16 21:09:46.324485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.323755 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-var-lib-cni-multus\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.324485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05328ec6-9d7f-41fa-b8a2-733ff3025de8-multus-daemon-config\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.324485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-sys-fs\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.324485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lsf8\" (UniqueName: \"kubernetes.io/projected/3a13edcb-9f53-409a-860f-0d1275da1cc4-kube-api-access-7lsf8\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.324485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-run\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.324485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-tuned\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.324485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324388 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-multus-cni-dir\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.324485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324411 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-os-release\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.324485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-multus-socket-dir-parent\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.324485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-run-multus-certs\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21f02b78-840d-4d87-a3da-ef3a063424a2-agent-certs\") pod \"konnectivity-agent-s2pdw\" (UID: \"21f02b78-840d-4d87-a3da-ef3a063424a2\") " pod="kube-system/konnectivity-agent-s2pdw" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d14ceb5-180d-4cd4-9653-70954a4f935b-host\") pod \"node-ca-bcb4c\" (UID: \"1d14ceb5-180d-4cd4-9653-70954a4f935b\") " pod="openshift-image-registry/node-ca-bcb4c" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-modprobe-d\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324574 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-lib-modules\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-var-lib-cni-bin\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-hostroot\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-socket-dir\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324657 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324679 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/244e9d5d-772a-4da3-8552-8e1c42165724-cni-binary-copy\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324701 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-systemd-units\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324715 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1ae55087-4ec5-476b-a36e-a46fc3ae4479-ovnkube-script-lib\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d14ceb5-180d-4cd4-9653-70954a4f935b-serviceca\") pod \"node-ca-bcb4c\" (UID: \"1d14ceb5-180d-4cd4-9653-70954a4f935b\") " pod="openshift-image-registry/node-ca-bcb4c" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-multus-conf-dir\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324784 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmkcr\" (UniqueName: \"kubernetes.io/projected/1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3-kube-api-access-lmkcr\") pod \"iptables-alerter-k44hd\" (UID: \"1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3\") " pod="openshift-network-operator/iptables-alerter-k44hd" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65dl8\" (UniqueName: \"kubernetes.io/projected/9a4e202f-99dc-4068-9496-acb28f0552b5-kube-api-access-65dl8\") pod \"network-metrics-daemon-twm75\" (UID: \"9a4e202f-99dc-4068-9496-acb28f0552b5\") " pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:09:46.324935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324803 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d14ceb5-180d-4cd4-9653-70954a4f935b-host\") pod \"node-ca-bcb4c\" (UID: \"1d14ceb5-180d-4cd4-9653-70954a4f935b\") " pod="openshift-image-registry/node-ca-bcb4c" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324831 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1ae55087-4ec5-476b-a36e-a46fc3ae4479-ovnkube-config\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324852 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05328ec6-9d7f-41fa-b8a2-733ff3025de8-cni-binary-copy\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324872 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret\") pod \"global-pull-secret-syncer-2xbp6\" (UID: \"3bd19c30-8019-4896-aa14-62214c3e042c\") " pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324877 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-sys-fs\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/244e9d5d-772a-4da3-8552-8e1c42165724-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-log-socket\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324945 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-var-lib-kubelet\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324971 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-systemd\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.324994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-sys\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325018 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-tmp\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325044 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/244e9d5d-772a-4da3-8552-8e1c42165724-cnibin\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325067 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/244e9d5d-772a-4da3-8552-8e1c42165724-os-release\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325170 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-run\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325173 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-modprobe-d\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325220 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-run-openvswitch\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325246 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-run-k8s-cni-cncf-io\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.325715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-registration-dir\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325286 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-lib-modules\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325319 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-device-dir\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325339 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-host\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325343 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-var-lib-cni-bin\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-var-lib-openvswitch\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325385 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-hostroot\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-run-ovn\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325421 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-run-ovn\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325421 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1ae55087-4ec5-476b-a36e-a46fc3ae4479-env-overrides\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-cnibin\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3bd19c30-8019-4896-aa14-62214c3e042c-kubelet-config\") pod \"global-pull-secret-syncer-2xbp6\" (UID: \"3bd19c30-8019-4896-aa14-62214c3e042c\") " pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs\") pod \"network-metrics-daemon-twm75\" (UID: \"9a4e202f-99dc-4068-9496-acb28f0552b5\") " pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325542 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/244e9d5d-772a-4da3-8552-8e1c42165724-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325585 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21f02b78-840d-4d87-a3da-ef3a063424a2-konnectivity-ca\") pod \"konnectivity-agent-s2pdw\" (UID: \"21f02b78-840d-4d87-a3da-ef3a063424a2\") " pod="kube-system/konnectivity-agent-s2pdw" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325615 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/244e9d5d-772a-4da3-8552-8e1c42165724-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.326522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-run-netns\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-run-systemd\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-cni-bin\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325722 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glcm4\" (UniqueName: \"kubernetes.io/projected/1ae55087-4ec5-476b-a36e-a46fc3ae4479-kube-api-access-glcm4\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-sysctl-conf\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325775 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-slash\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325786 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1ae55087-4ec5-476b-a36e-a46fc3ae4479-env-overrides\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325800 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-etc-kubernetes\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3-iptables-alerter-script\") pod \"iptables-alerter-k44hd\" (UID: \"1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3\") " pod="openshift-network-operator/iptables-alerter-k44hd" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325842 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-etc-kubernetes\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325873 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-sysconfig\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-var-lib-kubelet\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eda93f33-be42-4a44-ab7c-1c1e93d65b74-tmp-dir\") pod \"node-resolver-bb7qw\" (UID: \"eda93f33-be42-4a44-ab7c-1c1e93d65b74\") " pod="openshift-dns/node-resolver-bb7qw" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325955 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/244e9d5d-772a-4da3-8552-8e1c42165724-system-cni-dir\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.325982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-etc-openvswitch\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.326010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-system-cni-dir\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.326034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3bd19c30-8019-4896-aa14-62214c3e042c-dbus\") pod \"global-pull-secret-syncer-2xbp6\" (UID: \"3bd19c30-8019-4896-aa14-62214c3e042c\") " pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:46.327327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.326059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58stf\" (UniqueName: \"kubernetes.io/projected/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-kube-api-access-58stf\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.326086 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vzsk\" (UniqueName: \"kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk\") pod \"network-check-target-dbknf\" (UID: \"d278255e-2c35-4471-bbd9-095d9c476896\") " pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.326116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-run-ovn-kubernetes\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.326141 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-run-netns\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.326168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxshd\" (UniqueName: \"kubernetes.io/projected/eda93f33-be42-4a44-ab7c-1c1e93d65b74-kube-api-access-rxshd\") pod \"node-resolver-bb7qw\" (UID: \"eda93f33-be42-4a44-ab7c-1c1e93d65b74\") " pod="openshift-dns/node-resolver-bb7qw" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.326212 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-var-lib-kubelet\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.326316 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-socket-dir\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.326394 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.326456 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-systemd-units\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.326457 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-systemd\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.326498 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-sys\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.326513 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.326873 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-etc-openvswitch\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.326965 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d14ceb5-180d-4cd4-9653-70954a4f935b-serviceca\") pod \"node-ca-bcb4c\" (UID: \"1d14ceb5-180d-4cd4-9653-70954a4f935b\") " pod="openshift-image-registry/node-ca-bcb4c" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.327026 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-multus-conf-dir\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.327397 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3-iptables-alerter-script\") pod \"iptables-alerter-k44hd\" (UID: \"1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3\") " pod="openshift-network-operator/iptables-alerter-k44hd" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.327461 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-sysconfig\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.328151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.327505 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-var-lib-kubelet\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.327752 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eda93f33-be42-4a44-ab7c-1c1e93d65b74-tmp-dir\") pod \"node-resolver-bb7qw\" (UID: \"eda93f33-be42-4a44-ab7c-1c1e93d65b74\") " pod="openshift-dns/node-resolver-bb7qw" Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:46.327852 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.327868 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1ae55087-4ec5-476b-a36e-a46fc3ae4479-ovnkube-config\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.327901 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1ae55087-4ec5-476b-a36e-a46fc3ae4479-ovn-node-metrics-cert\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.327959 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-log-socket\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:46.327965 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret podName:3bd19c30-8019-4896-aa14-62214c3e042c nodeName:}" failed. No retries permitted until 2026-07-16 21:09:46.827944904 +0000 UTC m=+3.003053094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret") pod "global-pull-secret-syncer-2xbp6" (UID: "3bd19c30-8019-4896-aa14-62214c3e042c") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.328000 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-var-lib-openvswitch\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.328110 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-multus-cni-dir\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.328209 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-system-cni-dir\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.328209 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-multus-socket-dir-parent\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.328258 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-run-multus-certs\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.328270 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-os-release\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.328324 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3bd19c30-8019-4896-aa14-62214c3e042c-dbus\") pod \"global-pull-secret-syncer-2xbp6\" (UID: \"3bd19c30-8019-4896-aa14-62214c3e042c\") " pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.328370 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.328405 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-run-openvswitch\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.328457 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-run-k8s-cni-cncf-io\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.328502 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-run-ovn-kubernetes\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.328956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.328526 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05328ec6-9d7f-41fa-b8a2-733ff3025de8-multus-daemon-config\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.329815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.328570 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-cnibin\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.329815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.328626 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3bd19c30-8019-4896-aa14-62214c3e042c-kubelet-config\") pod \"global-pull-secret-syncer-2xbp6\" (UID: \"3bd19c30-8019-4896-aa14-62214c3e042c\") " pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:46.329815 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:46.328700 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:09:46.329815 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:46.328745 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs podName:9a4e202f-99dc-4068-9496-acb28f0552b5 nodeName:}" failed. No retries permitted until 2026-07-16 21:09:46.828730249 +0000 UTC m=+3.003838435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs") pod "network-metrics-daemon-twm75" (UID: "9a4e202f-99dc-4068-9496-acb28f0552b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:09:46.329815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.328924 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05328ec6-9d7f-41fa-b8a2-733ff3025de8-cni-binary-copy\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.329815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.328996 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-cni-bin\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.329815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.329042 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-run-netns\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.329815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.329116 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-run-systemd\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.329815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.329254 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-sysctl-conf\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.329815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.329257 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21f02b78-840d-4d87-a3da-ef3a063424a2-konnectivity-ca\") pod \"konnectivity-agent-s2pdw\" (UID: \"21f02b78-840d-4d87-a3da-ef3a063424a2\") " pod="kube-system/konnectivity-agent-s2pdw" Jul 16 21:09:46.329815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.329305 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ae55087-4ec5-476b-a36e-a46fc3ae4479-host-slash\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.329815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.329321 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05328ec6-9d7f-41fa-b8a2-733ff3025de8-host-run-netns\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.329815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.329345 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-device-dir\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.329815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.329355 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a13edcb-9f53-409a-860f-0d1275da1cc4-registration-dir\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.329815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.329368 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-host\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.331601 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.331558 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21f02b78-840d-4d87-a3da-ef3a063424a2-agent-certs\") pod \"konnectivity-agent-s2pdw\" (UID: \"21f02b78-840d-4d87-a3da-ef3a063424a2\") " pod="kube-system/konnectivity-agent-s2pdw" Jul 16 21:09:46.331697 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.331626 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-tmp\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.332319 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.332301 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-etc-tuned\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.338355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.338298 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmkcr\" (UniqueName: \"kubernetes.io/projected/1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3-kube-api-access-lmkcr\") pod \"iptables-alerter-k44hd\" (UID: \"1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3\") " pod="openshift-network-operator/iptables-alerter-k44hd" Jul 16 21:09:46.345784 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.345745 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lsf8\" (UniqueName: \"kubernetes.io/projected/3a13edcb-9f53-409a-860f-0d1275da1cc4-kube-api-access-7lsf8\") pod \"aws-ebs-csi-driver-node-xv6cd\" (UID: \"3a13edcb-9f53-409a-860f-0d1275da1cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.345970 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.345950 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58stf\" (UniqueName: \"kubernetes.io/projected/128787a6-a531-4f4b-85bc-6e5e11d7f8f5-kube-api-access-58stf\") pod \"tuned-2nd6g\" (UID: \"128787a6-a531-4f4b-85bc-6e5e11d7f8f5\") " pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.346197 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.346165 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glcm4\" (UniqueName: \"kubernetes.io/projected/1ae55087-4ec5-476b-a36e-a46fc3ae4479-kube-api-access-glcm4\") pod \"ovnkube-node-d5lc2\" (UID: \"1ae55087-4ec5-476b-a36e-a46fc3ae4479\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.346606 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.346576 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxshd\" (UniqueName: \"kubernetes.io/projected/eda93f33-be42-4a44-ab7c-1c1e93d65b74-kube-api-access-rxshd\") pod \"node-resolver-bb7qw\" (UID: \"eda93f33-be42-4a44-ab7c-1c1e93d65b74\") " pod="openshift-dns/node-resolver-bb7qw" Jul 16 21:09:46.346999 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.346971 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr87w\" (UniqueName: \"kubernetes.io/projected/1d14ceb5-180d-4cd4-9653-70954a4f935b-kube-api-access-tr87w\") pod \"node-ca-bcb4c\" (UID: \"1d14ceb5-180d-4cd4-9653-70954a4f935b\") " pod="openshift-image-registry/node-ca-bcb4c" Jul 16 21:09:46.347522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.347492 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvbdv\" (UniqueName: \"kubernetes.io/projected/05328ec6-9d7f-41fa-b8a2-733ff3025de8-kube-api-access-hvbdv\") pod \"multus-smplh\" (UID: \"05328ec6-9d7f-41fa-b8a2-733ff3025de8\") " pod="openshift-multus/multus-smplh" Jul 16 21:09:46.350629 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.350135 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65dl8\" (UniqueName: \"kubernetes.io/projected/9a4e202f-99dc-4068-9496-acb28f0552b5-kube-api-access-65dl8\") pod \"network-metrics-daemon-twm75\" (UID: \"9a4e202f-99dc-4068-9496-acb28f0552b5\") " pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:09:46.429339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.427204 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/244e9d5d-772a-4da3-8552-8e1c42165724-system-cni-dir\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.429339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.427263 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzsk\" (UniqueName: \"kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk\") pod \"network-check-target-dbknf\" (UID: \"d278255e-2c35-4471-bbd9-095d9c476896\") " pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:09:46.429339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.427294 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/244e9d5d-772a-4da3-8552-8e1c42165724-system-cni-dir\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.429339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.427303 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z888n\" (UniqueName: \"kubernetes.io/projected/244e9d5d-772a-4da3-8552-8e1c42165724-kube-api-access-z888n\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.429339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.427384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/244e9d5d-772a-4da3-8552-8e1c42165724-cni-binary-copy\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.429339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.427447 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/244e9d5d-772a-4da3-8552-8e1c42165724-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.429339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.427478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/244e9d5d-772a-4da3-8552-8e1c42165724-cnibin\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.429339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.427532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/244e9d5d-772a-4da3-8552-8e1c42165724-os-release\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.429339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.427591 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/244e9d5d-772a-4da3-8552-8e1c42165724-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.429339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.427628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/244e9d5d-772a-4da3-8552-8e1c42165724-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.429339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.427856 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/244e9d5d-772a-4da3-8552-8e1c42165724-cnibin\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.429339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.428285 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/244e9d5d-772a-4da3-8552-8e1c42165724-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.429339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.428376 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/244e9d5d-772a-4da3-8552-8e1c42165724-cni-binary-copy\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.429339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.428381 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/244e9d5d-772a-4da3-8552-8e1c42165724-os-release\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.429339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.428517 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/244e9d5d-772a-4da3-8552-8e1c42165724-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.429339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.428892 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/244e9d5d-772a-4da3-8552-8e1c42165724-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.446059 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:46.445497 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:09:46.446059 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:46.445528 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:09:46.446059 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:46.445545 2577 projected.go:196] Error preparing data for projected volume kube-api-access-5vzsk for pod openshift-network-diagnostics/network-check-target-dbknf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:09:46.446059 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:46.445608 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk podName:d278255e-2c35-4471-bbd9-095d9c476896 nodeName:}" failed. No retries permitted until 2026-07-16 21:09:46.945589523 +0000 UTC m=+3.120697709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5vzsk" (UniqueName: "kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk") pod "network-check-target-dbknf" (UID: "d278255e-2c35-4471-bbd9-095d9c476896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:09:46.446059 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.445970 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z888n\" (UniqueName: \"kubernetes.io/projected/244e9d5d-772a-4da3-8552-8e1c42165724-kube-api-access-z888n\") pod \"multus-additional-cni-plugins-gm6dn\" (UID: \"244e9d5d-772a-4da3-8552-8e1c42165724\") " pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.449782 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.449665 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-133.ec2.internal" event={"ID":"cd1e9b97c4c8ceff5fa6e561d0396716","Type":"ContainerStarted","Data":"b8995ab056102a597e6c7c8ab9b487c620d4c69ce6c81af18e9eaa8df301665c"} Jul 16 21:09:46.451220 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.451144 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal" event={"ID":"86c218ba58dac56ce79b99c5b33e3622","Type":"ContainerStarted","Data":"3272cd96fafac8847012d22f466eca602a110d391548c0f3fcd3b34f58a63a61"} Jul 16 21:09:46.511579 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.511357 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s2pdw" Jul 16 21:09:46.522157 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.521956 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:09:46.522614 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.522409 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:09:46.530925 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.530898 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bcb4c" Jul 16 21:09:46.537385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.537362 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-smplh" Jul 16 21:09:46.547343 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.547319 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k44hd" Jul 16 21:09:46.557071 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.557051 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" Jul 16 21:09:46.564786 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.564760 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bb7qw" Jul 16 21:09:46.574586 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.574563 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" Jul 16 21:09:46.580378 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.580353 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gm6dn" Jul 16 21:09:46.636294 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:46.634637 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128787a6_a531_4f4b_85bc_6e5e11d7f8f5.slice/crio-ce3745d587e4b1504c3bcd3dd966ba2691b2d71cd515452554e9e1867f77a066 WatchSource:0}: Error finding container ce3745d587e4b1504c3bcd3dd966ba2691b2d71cd515452554e9e1867f77a066: Status 404 returned error can't find the container with id ce3745d587e4b1504c3bcd3dd966ba2691b2d71cd515452554e9e1867f77a066 Jul 16 21:09:46.650983 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:46.650957 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d25258b_6d75_4758_9fbc_1a5b1f6a6ef3.slice/crio-70ec982fce42b0b20eaba4a52c274115bf486ec3e6882f5790fc5610b9f047d0 WatchSource:0}: Error finding container 70ec982fce42b0b20eaba4a52c274115bf486ec3e6882f5790fc5610b9f047d0: Status 404 returned error can't find the container with id 70ec982fce42b0b20eaba4a52c274115bf486ec3e6882f5790fc5610b9f047d0 Jul 16 21:09:46.651996 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:46.651971 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a13edcb_9f53_409a_860f_0d1275da1cc4.slice/crio-c3db1af4c4ba50ae142dfb86308c0e68fda4f02d1ea6db9bf58f2f22ff074d7c WatchSource:0}: Error finding container c3db1af4c4ba50ae142dfb86308c0e68fda4f02d1ea6db9bf58f2f22ff074d7c: Status 404 returned error can't find the container with id c3db1af4c4ba50ae142dfb86308c0e68fda4f02d1ea6db9bf58f2f22ff074d7c Jul 16 21:09:46.652244 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:09:46.652224 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d14ceb5_180d_4cd4_9653_70954a4f935b.slice/crio-1146039af42723df1117c66339a45f59883f32097153dfb60625b10181a5cb8f WatchSource:0}: Error finding container 1146039af42723df1117c66339a45f59883f32097153dfb60625b10181a5cb8f: Status 404 returned error can't find the container with id 1146039af42723df1117c66339a45f59883f32097153dfb60625b10181a5cb8f Jul 16 21:09:46.830555 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.830523 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret\") pod \"global-pull-secret-syncer-2xbp6\" (UID: \"3bd19c30-8019-4896-aa14-62214c3e042c\") " pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:46.830686 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:46.830571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs\") pod \"network-metrics-daemon-twm75\" (UID: \"9a4e202f-99dc-4068-9496-acb28f0552b5\") " pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:09:46.830758 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:46.830688 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:09:46.830758 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:46.830748 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs podName:9a4e202f-99dc-4068-9496-acb28f0552b5 nodeName:}" failed. No retries permitted until 2026-07-16 21:09:47.830733692 +0000 UTC m=+4.005841875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs") pod "network-metrics-daemon-twm75" (UID: "9a4e202f-99dc-4068-9496-acb28f0552b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:09:46.830853 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:46.830688 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:09:46.830853 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:46.830832 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret podName:3bd19c30-8019-4896-aa14-62214c3e042c nodeName:}" failed. No retries permitted until 2026-07-16 21:09:47.830818287 +0000 UTC m=+4.005926461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret") pod "global-pull-secret-syncer-2xbp6" (UID: "3bd19c30-8019-4896-aa14-62214c3e042c") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:09:47.031910 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.031877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzsk\" (UniqueName: \"kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk\") pod \"network-check-target-dbknf\" (UID: \"d278255e-2c35-4471-bbd9-095d9c476896\") " pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:09:47.032072 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:47.032001 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:09:47.032072 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:47.032017 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:09:47.032072 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:47.032026 2577 projected.go:196] Error preparing data for projected volume kube-api-access-5vzsk for pod openshift-network-diagnostics/network-check-target-dbknf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:09:47.032177 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:47.032078 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk podName:d278255e-2c35-4471-bbd9-095d9c476896 nodeName:}" failed. No retries permitted until 2026-07-16 21:09:48.032063971 +0000 UTC m=+4.207172143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vzsk" (UniqueName: "kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk") pod "network-check-target-dbknf" (UID: "d278255e-2c35-4471-bbd9-095d9c476896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:09:47.281198 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.281141 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:04:46 +0000 UTC" deadline="2028-03-27 01:18:45.785351724 +0000 UTC" Jul 16 21:09:47.281198 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.281196 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14860h8m58.504174888s" Jul 16 21:09:47.444705 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.444539 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:09:47.444884 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:47.444858 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:09:47.462212 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.461098 2577 generic.go:356] "Generic (PLEG): container finished" podID="86c218ba58dac56ce79b99c5b33e3622" containerID="71fb3ecd4b84ca43d16e36e2081d9d0d0d8e1973981029a3aff390945254929d" exitCode=0 Jul 16 21:09:47.462212 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.461194 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal" event={"ID":"86c218ba58dac56ce79b99c5b33e3622","Type":"ContainerDied","Data":"71fb3ecd4b84ca43d16e36e2081d9d0d0d8e1973981029a3aff390945254929d"} Jul 16 21:09:47.464934 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.464155 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" event={"ID":"3a13edcb-9f53-409a-860f-0d1275da1cc4","Type":"ContainerStarted","Data":"c3db1af4c4ba50ae142dfb86308c0e68fda4f02d1ea6db9bf58f2f22ff074d7c"} Jul 16 21:09:47.468140 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.468039 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" event={"ID":"1ae55087-4ec5-476b-a36e-a46fc3ae4479","Type":"ContainerStarted","Data":"0856b48d496af7992432b51f08a2b8e573993cfe0fa379f354bb0a1a77aca606"} Jul 16 21:09:47.473497 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.473438 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s2pdw" event={"ID":"21f02b78-840d-4d87-a3da-ef3a063424a2","Type":"ContainerStarted","Data":"a3d20362ff316a1c06d56671e7a5f8942faf74fb25f58e15cffda9a0b8876758"} Jul 16 21:09:47.475890 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.475840 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" event={"ID":"128787a6-a531-4f4b-85bc-6e5e11d7f8f5","Type":"ContainerStarted","Data":"ce3745d587e4b1504c3bcd3dd966ba2691b2d71cd515452554e9e1867f77a066"} Jul 16 21:09:47.479045 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.478492 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bcb4c" event={"ID":"1d14ceb5-180d-4cd4-9653-70954a4f935b","Type":"ContainerStarted","Data":"1146039af42723df1117c66339a45f59883f32097153dfb60625b10181a5cb8f"} Jul 16 21:09:47.480394 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.480332 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k44hd" event={"ID":"1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3","Type":"ContainerStarted","Data":"70ec982fce42b0b20eaba4a52c274115bf486ec3e6882f5790fc5610b9f047d0"} Jul 16 21:09:47.483300 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.483208 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bb7qw" event={"ID":"eda93f33-be42-4a44-ab7c-1c1e93d65b74","Type":"ContainerStarted","Data":"a269f3a21584912b6514d633e3766fa7cb07cc90b9308fca71b0c3e84ff4629c"} Jul 16 21:09:47.486218 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.486142 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm6dn" event={"ID":"244e9d5d-772a-4da3-8552-8e1c42165724","Type":"ContainerStarted","Data":"90b9c88d61de20af23f89ed2411cf0a9ebe0c7b23ed2b8aac09eedab9b3f5e5a"} Jul 16 21:09:47.490696 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.490638 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-smplh" event={"ID":"05328ec6-9d7f-41fa-b8a2-733ff3025de8","Type":"ContainerStarted","Data":"f05c9547a2b20a3494a363714d60a9843149abe6871ff2439ab9138d412d4df4"} Jul 16 21:09:47.839090 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.839054 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret\") pod \"global-pull-secret-syncer-2xbp6\" (UID: \"3bd19c30-8019-4896-aa14-62214c3e042c\") " pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:47.839299 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:47.839125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs\") pod \"network-metrics-daemon-twm75\" (UID: \"9a4e202f-99dc-4068-9496-acb28f0552b5\") " pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:09:47.839372 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:47.839308 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:09:47.839431 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:47.839379 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs podName:9a4e202f-99dc-4068-9496-acb28f0552b5 nodeName:}" failed. No retries permitted until 2026-07-16 21:09:49.839359241 +0000 UTC m=+6.014467417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs") pod "network-metrics-daemon-twm75" (UID: "9a4e202f-99dc-4068-9496-acb28f0552b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:09:47.839948 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:47.839608 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:09:47.839948 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:47.839674 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret podName:3bd19c30-8019-4896-aa14-62214c3e042c nodeName:}" failed. No retries permitted until 2026-07-16 21:09:49.839655558 +0000 UTC m=+6.014763733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret") pod "global-pull-secret-syncer-2xbp6" (UID: "3bd19c30-8019-4896-aa14-62214c3e042c") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:09:48.041360 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:48.041323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzsk\" (UniqueName: \"kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk\") pod \"network-check-target-dbknf\" (UID: \"d278255e-2c35-4471-bbd9-095d9c476896\") " pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:09:48.041532 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:48.041488 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:09:48.041532 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:48.041506 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:09:48.041532 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:48.041518 2577 projected.go:196] Error preparing data for projected volume kube-api-access-5vzsk for pod openshift-network-diagnostics/network-check-target-dbknf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:09:48.041688 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:48.041573 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk podName:d278255e-2c35-4471-bbd9-095d9c476896 nodeName:}" failed. No retries permitted until 2026-07-16 21:09:50.04155509 +0000 UTC m=+6.216663277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vzsk" (UniqueName: "kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk") pod "network-check-target-dbknf" (UID: "d278255e-2c35-4471-bbd9-095d9c476896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:09:48.281820 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:48.281706 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:04:46 +0000 UTC" deadline="2027-12-26 18:58:12.958143732 +0000 UTC" Jul 16 21:09:48.281820 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:48.281769 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12669h48m24.676378717s" Jul 16 21:09:48.444594 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:48.444556 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:48.444768 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:48.444580 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:09:48.444768 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:48.444705 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:09:48.444768 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:48.444737 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:09:48.499039 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:48.498376 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-133.ec2.internal" event={"ID":"cd1e9b97c4c8ceff5fa6e561d0396716","Type":"ContainerStarted","Data":"fc2fe1b8544724e0da88a69fccb9fac49398dddbdbd225b21ae586932c6ffbcd"} Jul 16 21:09:48.505992 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:48.505961 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal" event={"ID":"86c218ba58dac56ce79b99c5b33e3622","Type":"ContainerStarted","Data":"16190dc2942f5a85f0f19b3e288bdf9d167a1bc9f6767b955dafcd4bee4e6489"} Jul 16 21:09:48.536617 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:48.536499 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-133.ec2.internal" podStartSLOduration=3.536456189 podStartE2EDuration="3.536456189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:09:48.516594698 +0000 UTC m=+4.691702911" watchObservedRunningTime="2026-07-16 21:09:48.536456189 +0000 UTC m=+4.711564384" Jul 16 21:09:49.445123 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:49.445090 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:09:49.445596 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:49.445237 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:09:49.858636 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:49.858005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret\") pod \"global-pull-secret-syncer-2xbp6\" (UID: \"3bd19c30-8019-4896-aa14-62214c3e042c\") " pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:49.858636 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:49.858066 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs\") pod \"network-metrics-daemon-twm75\" (UID: \"9a4e202f-99dc-4068-9496-acb28f0552b5\") " pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:09:49.858636 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:49.858203 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:09:49.858636 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:49.858229 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:09:49.858636 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:49.858265 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs podName:9a4e202f-99dc-4068-9496-acb28f0552b5 nodeName:}" failed. No retries permitted until 2026-07-16 21:09:53.858247145 +0000 UTC m=+10.033355327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs") pod "network-metrics-daemon-twm75" (UID: "9a4e202f-99dc-4068-9496-acb28f0552b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:09:49.858636 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:49.858294 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret podName:3bd19c30-8019-4896-aa14-62214c3e042c nodeName:}" failed. No retries permitted until 2026-07-16 21:09:53.858277324 +0000 UTC m=+10.033385497 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret") pod "global-pull-secret-syncer-2xbp6" (UID: "3bd19c30-8019-4896-aa14-62214c3e042c") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:09:50.059376 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:50.059345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzsk\" (UniqueName: \"kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk\") pod \"network-check-target-dbknf\" (UID: \"d278255e-2c35-4471-bbd9-095d9c476896\") " pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:09:50.059556 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:50.059506 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:09:50.059556 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:50.059533 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:09:50.059556 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:50.059547 2577 projected.go:196] Error preparing data for projected volume kube-api-access-5vzsk for pod openshift-network-diagnostics/network-check-target-dbknf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:09:50.059717 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:50.059609 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk podName:d278255e-2c35-4471-bbd9-095d9c476896 nodeName:}" failed. No retries permitted until 2026-07-16 21:09:54.059588933 +0000 UTC m=+10.234697118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vzsk" (UniqueName: "kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk") pod "network-check-target-dbknf" (UID: "d278255e-2c35-4471-bbd9-095d9c476896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:09:50.445480 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:50.445447 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:50.446121 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:50.445573 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:09:50.446121 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:50.446060 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:09:50.446256 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:50.446173 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:09:51.444817 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:51.444744 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:09:51.445000 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:51.444920 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:09:52.444914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:52.444879 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:09:52.445374 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:52.445024 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:09:52.445374 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:52.445068 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:52.445374 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:52.445167 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:09:53.444905 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:53.444791 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:09:53.445078 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:53.444923 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:09:53.892734 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:53.892697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret\") pod \"global-pull-secret-syncer-2xbp6\" (UID: \"3bd19c30-8019-4896-aa14-62214c3e042c\") " pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:53.892920 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:53.892760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs\") pod \"network-metrics-daemon-twm75\" (UID: \"9a4e202f-99dc-4068-9496-acb28f0552b5\") " pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:09:53.892920 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:53.892879 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:09:53.893051 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:53.892963 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret podName:3bd19c30-8019-4896-aa14-62214c3e042c nodeName:}" failed. No retries permitted until 2026-07-16 21:10:01.892944921 +0000 UTC m=+18.068053316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret") pod "global-pull-secret-syncer-2xbp6" (UID: "3bd19c30-8019-4896-aa14-62214c3e042c") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:09:53.893051 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:53.892886 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:09:53.893161 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:53.893073 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs podName:9a4e202f-99dc-4068-9496-acb28f0552b5 nodeName:}" failed. No retries permitted until 2026-07-16 21:10:01.893020454 +0000 UTC m=+18.068128626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs") pod "network-metrics-daemon-twm75" (UID: "9a4e202f-99dc-4068-9496-acb28f0552b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:09:54.094422 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:54.094382 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzsk\" (UniqueName: \"kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk\") pod \"network-check-target-dbknf\" (UID: \"d278255e-2c35-4471-bbd9-095d9c476896\") " pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:09:54.094635 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:54.094549 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:09:54.094715 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:54.094645 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:09:54.094715 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:54.094660 2577 projected.go:196] Error preparing data for projected volume kube-api-access-5vzsk for pod openshift-network-diagnostics/network-check-target-dbknf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:09:54.094814 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:54.094723 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk podName:d278255e-2c35-4471-bbd9-095d9c476896 nodeName:}" failed. No retries permitted until 2026-07-16 21:10:02.094705313 +0000 UTC m=+18.269813500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vzsk" (UniqueName: "kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk") pod "network-check-target-dbknf" (UID: "d278255e-2c35-4471-bbd9-095d9c476896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:09:54.445203 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:54.445161 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:54.445559 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:54.445305 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:09:54.446830 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:54.446805 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:09:54.446965 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:54.446924 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:09:55.444889 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:55.444707 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:09:55.444889 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:55.444835 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:09:56.445556 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:56.445520 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:56.446004 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:56.445521 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:09:56.446004 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:56.445660 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:09:56.446004 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:56.445764 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:09:57.444567 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:57.444540 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:09:57.444742 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:57.444654 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:09:58.445226 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:58.445176 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:09:58.445694 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:58.445333 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:09:58.445694 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:58.445383 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:09:58.445694 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:58.445484 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:09:59.445446 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:09:59.445413 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:09:59.445824 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:09:59.445531 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:10:00.445267 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:00.445227 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:10:00.445440 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:00.445356 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:10:00.445440 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:00.445417 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:10:00.445778 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:00.445550 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:10:01.445068 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:01.445038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:01.445307 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:01.445150 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:10:01.958060 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:01.958028 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret\") pod \"global-pull-secret-syncer-2xbp6\" (UID: \"3bd19c30-8019-4896-aa14-62214c3e042c\") " pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:10:01.958487 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:01.958076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs\") pod \"network-metrics-daemon-twm75\" (UID: \"9a4e202f-99dc-4068-9496-acb28f0552b5\") " pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:10:01.958487 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:01.958207 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:10:01.958487 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:01.958216 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:10:01.958487 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:01.958273 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs podName:9a4e202f-99dc-4068-9496-acb28f0552b5 nodeName:}" failed. No retries permitted until 2026-07-16 21:10:17.958255037 +0000 UTC m=+34.133363221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs") pod "network-metrics-daemon-twm75" (UID: "9a4e202f-99dc-4068-9496-acb28f0552b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:10:01.958487 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:01.958291 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret podName:3bd19c30-8019-4896-aa14-62214c3e042c nodeName:}" failed. No retries permitted until 2026-07-16 21:10:17.95828287 +0000 UTC m=+34.133391039 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret") pod "global-pull-secret-syncer-2xbp6" (UID: "3bd19c30-8019-4896-aa14-62214c3e042c") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:10:02.159011 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:02.158981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzsk\" (UniqueName: \"kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk\") pod \"network-check-target-dbknf\" (UID: \"d278255e-2c35-4471-bbd9-095d9c476896\") " pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:02.159217 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:02.159128 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:10:02.159217 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:02.159149 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:10:02.159217 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:02.159162 2577 projected.go:196] Error preparing data for projected volume kube-api-access-5vzsk for pod openshift-network-diagnostics/network-check-target-dbknf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:10:02.159383 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:02.159237 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk podName:d278255e-2c35-4471-bbd9-095d9c476896 nodeName:}" failed. No retries permitted until 2026-07-16 21:10:18.159218277 +0000 UTC m=+34.334326460 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vzsk" (UniqueName: "kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk") pod "network-check-target-dbknf" (UID: "d278255e-2c35-4471-bbd9-095d9c476896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:10:02.444609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:02.444574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:10:02.444777 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:02.444588 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:10:02.444777 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:02.444725 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:10:02.444896 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:02.444782 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:10:03.445385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:03.445349 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:03.445801 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:03.445476 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:10:04.445853 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:04.445815 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:10:04.446297 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:04.445938 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:10:04.446297 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:04.445977 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:10:04.446297 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:04.446070 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:10:05.444723 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:05.444701 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:05.444807 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:05.444790 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:10:06.445368 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.445171 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:10:06.445806 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.445245 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:10:06.445806 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:06.445469 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:10:06.445806 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:06.445505 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:10:06.541391 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.541365 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" event={"ID":"3a13edcb-9f53-409a-860f-0d1275da1cc4","Type":"ContainerStarted","Data":"fddf79755f822f05106cd6b8eba3e160180f0f392c6d65abc7c3983f706fa969"} Jul 16 21:10:06.542815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.542797 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:10:06.543170 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.543152 2577 generic.go:356] "Generic (PLEG): container finished" podID="1ae55087-4ec5-476b-a36e-a46fc3ae4479" containerID="c3075f0f084a33f529356b644631f55ec5f246886ff6c95d98500addd7a5fa7f" exitCode=1 Jul 16 21:10:06.543278 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.543208 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" event={"ID":"1ae55087-4ec5-476b-a36e-a46fc3ae4479","Type":"ContainerDied","Data":"c3075f0f084a33f529356b644631f55ec5f246886ff6c95d98500addd7a5fa7f"} Jul 16 21:10:06.543278 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.543269 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" event={"ID":"1ae55087-4ec5-476b-a36e-a46fc3ae4479","Type":"ContainerStarted","Data":"b40adac4daf0917f23112b04539fb947f88f964f66ea74cfec8bfdc4cce0c60d"} Jul 16 21:10:06.544667 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.544647 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s2pdw" event={"ID":"21f02b78-840d-4d87-a3da-ef3a063424a2","Type":"ContainerStarted","Data":"6b31df55be105c68cdfca2e84ead85bcf111c221cce1742bfba56fefff31bb2a"} Jul 16 21:10:06.546174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.546155 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" event={"ID":"128787a6-a531-4f4b-85bc-6e5e11d7f8f5","Type":"ContainerStarted","Data":"a15782f5457ae7cd053320a3447e3eb3a2760b3fe2f052f0995a505103368565"} Jul 16 21:10:06.547602 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.547512 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bcb4c" event={"ID":"1d14ceb5-180d-4cd4-9653-70954a4f935b","Type":"ContainerStarted","Data":"7d49d0793d374fce5fa27462b0bb8e4cb9de1b428ca0272adef7bfcf542c55b7"} Jul 16 21:10:06.548976 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.548913 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bb7qw" event={"ID":"eda93f33-be42-4a44-ab7c-1c1e93d65b74","Type":"ContainerStarted","Data":"8e8a85acabd51124c8c09709db32e4d8ffeb335e19bc90b0954729e724e2df86"} Jul 16 21:10:06.550587 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.550564 2577 generic.go:356] "Generic (PLEG): container finished" podID="244e9d5d-772a-4da3-8552-8e1c42165724" containerID="d28085752c84386d86d86ed851ea0235f449c465746c3e5e455c85d61f877b43" exitCode=0 Jul 16 21:10:06.550676 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.550636 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm6dn" event={"ID":"244e9d5d-772a-4da3-8552-8e1c42165724","Type":"ContainerDied","Data":"d28085752c84386d86d86ed851ea0235f449c465746c3e5e455c85d61f877b43"} Jul 16 21:10:06.552177 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.552159 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-smplh" event={"ID":"05328ec6-9d7f-41fa-b8a2-733ff3025de8","Type":"ContainerStarted","Data":"05ae9344ee65c9b0af6cb5985aabd409fe76768c935d13bd078efcf97a1c048a"} Jul 16 21:10:06.567697 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.567660 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-133.ec2.internal" podStartSLOduration=21.56764977 podStartE2EDuration="21.56764977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:09:48.536988428 +0000 UTC m=+4.712096621" watchObservedRunningTime="2026-07-16 21:10:06.56764977 +0000 UTC m=+22.742757962" Jul 16 21:10:06.602343 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.602305 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s2pdw" podStartSLOduration=3.547970454 podStartE2EDuration="22.602289174s" totalImagesPullingTime="19.05431872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:06.567776282 +0000 UTC m=+22.742884465" watchObservedRunningTime="2026-07-16 21:10:06.602289174 +0000 UTC m=+22.777397424" Jul 16 21:10:06.624848 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.624677 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-smplh" podStartSLOduration=3.562288341 podStartE2EDuration="22.624663096s" totalImagesPullingTime="19.062374755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:06.602425008 +0000 UTC m=+22.777533190" watchObservedRunningTime="2026-07-16 21:10:06.624663096 +0000 UTC m=+22.799771287" Jul 16 21:10:06.651404 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.651353 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2nd6g" podStartSLOduration=3.6188171479999998 podStartE2EDuration="22.65133458s" totalImagesPullingTime="19.032517432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:06.625695601 +0000 UTC m=+22.800803816" watchObservedRunningTime="2026-07-16 21:10:06.65133458 +0000 UTC m=+22.826442773" Jul 16 21:10:06.719380 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:06.719334 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bcb4c" podStartSLOduration=3.703795123 podStartE2EDuration="22.719314577s" totalImagesPullingTime="19.015519454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:06.680243636 +0000 UTC m=+22.855351849" watchObservedRunningTime="2026-07-16 21:10:06.719314577 +0000 UTC m=+22.894422770" Jul 16 21:10:07.398418 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:07.398398 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 21:10:07.445231 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:07.445209 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:07.445353 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:07.445320 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:10:07.555259 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:07.555235 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k44hd" event={"ID":"1d25258b-6d75-4758-9fbc-1a5b1f6a6ef3","Type":"ContainerStarted","Data":"a1184833f70f9aaef23fe486371dbd162a91d221d16a5514dd0fd03bdcbd6c0a"} Jul 16 21:10:07.556995 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:07.556972 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" event={"ID":"3a13edcb-9f53-409a-860f-0d1275da1cc4","Type":"ContainerStarted","Data":"17e19bff30dbb6ff2afe003e227d0c4ec9d7de0e199b97da094a4bb1d59502c7"} Jul 16 21:10:07.559458 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:07.559437 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:10:07.559862 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:07.559843 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" event={"ID":"1ae55087-4ec5-476b-a36e-a46fc3ae4479","Type":"ContainerStarted","Data":"9c81b9afae1bbb5d389bb270f998d0693fe983fe8b8981001984226e1321297c"} Jul 16 21:10:07.559953 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:07.559918 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" event={"ID":"1ae55087-4ec5-476b-a36e-a46fc3ae4479","Type":"ContainerStarted","Data":"c656475768b2d6e41c91b4f6437b35ff56c5d0659d48f9bbe7a82d8227b7ed18"} Jul 16 21:10:07.560018 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:07.559972 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" event={"ID":"1ae55087-4ec5-476b-a36e-a46fc3ae4479","Type":"ContainerStarted","Data":"1136ea116095885a1a196fc3830c75b3a39f839fce8ceb9d1d27029ec79296ca"} Jul 16 21:10:07.560071 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:07.560024 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" event={"ID":"1ae55087-4ec5-476b-a36e-a46fc3ae4479","Type":"ContainerStarted","Data":"b95cfcce3e9c840ebdf888ec7de439379534b02db8eafbbb2ce3203538046662"} Jul 16 21:10:07.572612 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:07.572575 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bb7qw" podStartSLOduration=4.552128745 podStartE2EDuration="23.572562967s" totalImagesPullingTime="19.020434222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:06.71967862 +0000 UTC m=+22.894786813" watchObservedRunningTime="2026-07-16 21:10:07.572562967 +0000 UTC m=+23.747671155" Jul 16 21:10:07.584367 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:07.584344 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s2pdw" Jul 16 21:10:07.584884 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:07.584864 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s2pdw" Jul 16 21:10:07.609289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:07.609227 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k44hd" podStartSLOduration=4.562374769 podStartE2EDuration="23.609217793s" totalImagesPullingTime="19.046843024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:07.572206967 +0000 UTC m=+23.747315176" watchObservedRunningTime="2026-07-16 21:10:07.609217793 +0000 UTC m=+23.784325984" Jul 16 21:10:08.347106 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:08.346983 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T21:10:07.398414811Z","UUID":"42c0e9cc-9625-4e50-8765-379c1180d7d9","Handler":null,"Name":"","Endpoint":""} Jul 16 21:10:08.348811 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:08.348790 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 21:10:08.348944 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:08.348831 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 21:10:08.445418 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:08.445386 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:10:08.445590 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:08.445525 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:10:08.445914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:08.445761 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:10:08.445914 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:08.445877 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:10:08.562065 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:08.562019 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s2pdw" Jul 16 21:10:08.563030 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:08.562306 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s2pdw" Jul 16 21:10:09.444695 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:09.444425 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:09.444852 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:09.444758 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:10:09.566201 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:09.566147 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" event={"ID":"3a13edcb-9f53-409a-860f-0d1275da1cc4","Type":"ContainerStarted","Data":"03876b028941febd05fc040ce8a0b30dde84c08e2b211b7dc2e5d8bee0e11753"} Jul 16 21:10:09.569883 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:09.569861 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:10:09.570256 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:09.570227 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" event={"ID":"1ae55087-4ec5-476b-a36e-a46fc3ae4479","Type":"ContainerStarted","Data":"10ff787553bf85259d80f5aced5b67126796f7a45d614ee3a82a467624a696ef"} Jul 16 21:10:10.445011 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:10.444974 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:10:10.445237 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:10.444975 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:10:10.445237 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:10.445102 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:10:10.445237 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:10.445169 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:10:11.445012 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:11.444981 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:11.445513 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:11.445082 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:10:12.444812 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:12.444594 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:10:12.444907 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:12.444657 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:10:12.444907 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:12.444859 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:10:12.444991 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:12.444902 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:10:12.579285 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:12.579263 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:10:12.579886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:12.579651 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" event={"ID":"1ae55087-4ec5-476b-a36e-a46fc3ae4479","Type":"ContainerStarted","Data":"0774871f2e0eb50d9e91acf48ffd8860be5a1242581f9fd20da83710813d3a76"} Jul 16 21:10:12.579886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:12.579788 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:10:12.579886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:12.579859 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:10:12.580053 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:12.579913 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:10:12.580111 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:12.580074 2577 scope.go:117] "RemoveContainer" containerID="c3075f0f084a33f529356b644631f55ec5f246886ff6c95d98500addd7a5fa7f" Jul 16 21:10:12.581667 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:12.581644 2577 generic.go:356] "Generic (PLEG): container finished" podID="244e9d5d-772a-4da3-8552-8e1c42165724" containerID="40a85363e66b6e3dc4e374a1f444be20632c14abefad4bb645c845db7fae5ac9" exitCode=0 Jul 16 21:10:12.581752 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:12.581679 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm6dn" event={"ID":"244e9d5d-772a-4da3-8552-8e1c42165724","Type":"ContainerDied","Data":"40a85363e66b6e3dc4e374a1f444be20632c14abefad4bb645c845db7fae5ac9"} Jul 16 21:10:12.596551 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:12.596529 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:10:12.596665 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:12.596653 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:10:12.621569 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:12.621531 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xv6cd" podStartSLOduration=6.884768729 podStartE2EDuration="28.621519291s" totalImagesPullingTime="21.736750562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:09:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:09.595423973 +0000 UTC m=+25.770532165" watchObservedRunningTime="2026-07-16 21:10:12.621519291 +0000 UTC m=+28.796627482" Jul 16 21:10:13.444853 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:13.444826 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:13.444999 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:13.444918 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:10:13.589404 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:13.589379 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:10:13.589781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:13.589716 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" event={"ID":"1ae55087-4ec5-476b-a36e-a46fc3ae4479","Type":"ContainerStarted","Data":"d0de5e276df27176f04e2caed130b116dfbaf851787028be8dd0817c59223281"} Jul 16 21:10:13.634472 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:13.634421 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" podStartSLOduration=9.981023013 podStartE2EDuration="29.634405304s" totalImagesPullingTime="19.653382291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:13.630950897 +0000 UTC m=+29.806059115" watchObservedRunningTime="2026-07-16 21:10:13.634405304 +0000 UTC m=+29.809513523" Jul 16 21:10:14.375239 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:14.375153 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dbknf"] Jul 16 21:10:14.375351 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:14.375298 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:14.375393 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:14.375372 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:10:14.382249 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:14.382226 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-twm75"] Jul 16 21:10:14.382364 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:14.382319 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:10:14.382412 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:14.382396 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:10:14.397382 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:14.397346 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2xbp6"] Jul 16 21:10:14.397499 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:14.397436 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:10:14.397554 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:14.397513 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:10:14.594039 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:14.594002 2577 generic.go:356] "Generic (PLEG): container finished" podID="244e9d5d-772a-4da3-8552-8e1c42165724" containerID="b062aa268e3683ca304602dc03748520f01eb6d4a4bfb5d9499c814e62f63426" exitCode=0 Jul 16 21:10:14.594417 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:14.594089 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm6dn" event={"ID":"244e9d5d-772a-4da3-8552-8e1c42165724","Type":"ContainerDied","Data":"b062aa268e3683ca304602dc03748520f01eb6d4a4bfb5d9499c814e62f63426"} Jul 16 21:10:15.444789 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:15.444758 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:10:15.444908 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:15.444862 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:10:15.835459 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:15.835283 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bb7qw_eda93f33-be42-4a44-ab7c-1c1e93d65b74/dns-node-resolver/0.log" Jul 16 21:10:16.447957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:16.447931 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:16.448093 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:16.448029 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:10:16.448093 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:16.448083 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:10:16.448198 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:16.448168 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:10:16.600374 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:16.600338 2577 generic.go:356] "Generic (PLEG): container finished" podID="244e9d5d-772a-4da3-8552-8e1c42165724" containerID="f656a51cfe3e815c8fe607fee84f984dec6cf04397557cbc9b810c809df90f71" exitCode=0 Jul 16 21:10:16.600503 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:16.600392 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm6dn" event={"ID":"244e9d5d-772a-4da3-8552-8e1c42165724","Type":"ContainerDied","Data":"f656a51cfe3e815c8fe607fee84f984dec6cf04397557cbc9b810c809df90f71"} Jul 16 21:10:16.792988 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:16.792912 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bcb4c_1d14ceb5-180d-4cd4-9653-70954a4f935b/node-ca/0.log" Jul 16 21:10:17.445427 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:17.445351 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:10:17.445955 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:17.445464 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twm75" podUID="9a4e202f-99dc-4068-9496-acb28f0552b5" Jul 16 21:10:17.981623 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:17.981593 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret\") pod \"global-pull-secret-syncer-2xbp6\" (UID: \"3bd19c30-8019-4896-aa14-62214c3e042c\") " pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:10:17.981798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:17.981641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs\") pod \"network-metrics-daemon-twm75\" (UID: \"9a4e202f-99dc-4068-9496-acb28f0552b5\") " pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:10:17.981798 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:17.981770 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:10:17.981916 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:17.981838 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret podName:3bd19c30-8019-4896-aa14-62214c3e042c nodeName:}" failed. No retries permitted until 2026-07-16 21:10:49.981817399 +0000 UTC m=+66.156925573 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret") pod "global-pull-secret-syncer-2xbp6" (UID: "3bd19c30-8019-4896-aa14-62214c3e042c") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:10:17.981916 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:17.981843 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:10:17.981916 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:17.981890 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs podName:9a4e202f-99dc-4068-9496-acb28f0552b5 nodeName:}" failed. No retries permitted until 2026-07-16 21:10:49.981873532 +0000 UTC m=+66.156981717 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs") pod "network-metrics-daemon-twm75" (UID: "9a4e202f-99dc-4068-9496-acb28f0552b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:10:18.183009 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:18.182976 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzsk\" (UniqueName: \"kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk\") pod \"network-check-target-dbknf\" (UID: \"d278255e-2c35-4471-bbd9-095d9c476896\") " pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:18.183251 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:18.183232 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:10:18.183341 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:18.183257 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:10:18.183341 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:18.183270 2577 projected.go:196] Error preparing data for projected volume kube-api-access-5vzsk for pod openshift-network-diagnostics/network-check-target-dbknf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:10:18.183341 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:18.183333 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk podName:d278255e-2c35-4471-bbd9-095d9c476896 nodeName:}" failed. No retries permitted until 2026-07-16 21:10:50.183314867 +0000 UTC m=+66.358423052 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vzsk" (UniqueName: "kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk") pod "network-check-target-dbknf" (UID: "d278255e-2c35-4471-bbd9-095d9c476896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:10:18.448512 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:18.448485 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:18.448973 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:18.448492 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:10:18.448973 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:18.448609 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbknf" podUID="d278255e-2c35-4471-bbd9-095d9c476896" Jul 16 21:10:18.448973 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:18.448673 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xbp6" podUID="3bd19c30-8019-4896-aa14-62214c3e042c" Jul 16 21:10:18.576984 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:18.576907 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-133.ec2.internal" event="NodeReady" Jul 16 21:10:18.577130 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:18.577052 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 21:10:19.444578 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:19.444546 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:10:19.449281 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:19.449248 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 21:10:19.449281 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:19.449272 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ljktf\"" Jul 16 21:10:20.448642 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:20.448614 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:20.448824 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:20.448614 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:10:20.459010 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:20.458991 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 21:10:20.459860 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:20.459837 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xwmds\"" Jul 16 21:10:20.460517 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:20.460498 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 21:10:20.461266 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:20.461247 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 21:10:21.614407 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:21.614372 2577 generic.go:356] "Generic (PLEG): container finished" podID="244e9d5d-772a-4da3-8552-8e1c42165724" containerID="d598cc8ed283ed680fb3ea21ac72b294e8c3629945f7ee3f3704cd420cc20a74" exitCode=0 Jul 16 21:10:21.614857 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:21.614454 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm6dn" event={"ID":"244e9d5d-772a-4da3-8552-8e1c42165724","Type":"ContainerDied","Data":"d598cc8ed283ed680fb3ea21ac72b294e8c3629945f7ee3f3704cd420cc20a74"} Jul 16 21:10:22.619230 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:22.619176 2577 generic.go:356] "Generic (PLEG): container finished" podID="244e9d5d-772a-4da3-8552-8e1c42165724" containerID="ee2704e9b30cbe2a091a1803263acaefffc5dec0737488d8b77e5b6e9760af95" exitCode=0 Jul 16 21:10:22.619230 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:22.619219 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm6dn" event={"ID":"244e9d5d-772a-4da3-8552-8e1c42165724","Type":"ContainerDied","Data":"ee2704e9b30cbe2a091a1803263acaefffc5dec0737488d8b77e5b6e9760af95"} Jul 16 21:10:23.625258 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:23.625043 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm6dn" event={"ID":"244e9d5d-772a-4da3-8552-8e1c42165724","Type":"ContainerStarted","Data":"b1df624a3b44be228790bc6d17c7cf86d402efdeba399e7c6e7abeee0e20a41c"} Jul 16 21:10:23.707756 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:23.707714 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gm6dn" podStartSLOduration=8.145214332 podStartE2EDuration="39.707698273s" totalImagesPullingTime="31.562483941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:09:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:23.707619257 +0000 UTC m=+39.882727450" watchObservedRunningTime="2026-07-16 21:10:23.707698273 +0000 UTC m=+39.882806461" Jul 16 21:10:39.449033 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.448999 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cd4df479d-m2fwl"] Jul 16 21:10:39.451078 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.451062 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.462961 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.462915 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 21:10:39.467272 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.467251 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 21:10:39.467548 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.467532 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m22xl\"" Jul 16 21:10:39.467758 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.467746 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 21:10:39.468614 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.468599 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zzmpt"] Jul 16 21:10:39.470401 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.470385 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zzmpt" Jul 16 21:10:39.475217 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:39.475174 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-dockercfg-5s8t4\" is forbidden: User \"system:node:ip-10-0-137-133.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-137-133.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5s8t4\"" type="*v1.Secret" Jul 16 21:10:39.476452 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:39.476428 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-default-metrics-tls\" is forbidden: User \"system:node:ip-10-0-137-133.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-137-133.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" Jul 16 21:10:39.476452 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:39.476429 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"dns-default\" is forbidden: User \"system:node:ip-10-0-137-133.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-137-133.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" Jul 16 21:10:39.478111 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.478094 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 21:10:39.482365 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.482346 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pjlc6"] Jul 16 21:10:39.484178 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.484164 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:39.489145 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:39.489121 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-sa-dockercfg-glgkx\" is forbidden: User \"system:node:ip-10-0-137-133.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-137-133.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-glgkx\"" type="*v1.Secret" Jul 16 21:10:39.490095 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:39.490073 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-137-133.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-137-133.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 16 21:10:39.491973 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:39.491952 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-137-133.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-137-133.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" Jul 16 21:10:39.492059 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:39.491973 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-137-133.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-137-133.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 16 21:10:39.492902 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:39.492880 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-137-133.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-137-133.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" Jul 16 21:10:39.511910 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.511881 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zzmpt"] Jul 16 21:10:39.512022 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.511988 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cd4df479d-m2fwl"] Jul 16 21:10:39.519287 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.519267 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pjlc6"] Jul 16 21:10:39.522537 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.522519 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dx929"] Jul 16 21:10:39.524751 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.524735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dx929" Jul 16 21:10:39.529464 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529440 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4e81032-aa8b-4011-a300-24472e2b6ab1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:39.529551 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529477 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1934850b-46ad-4994-9166-22aa589e1a0d-image-registry-private-configuration\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.529551 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529509 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf6lq\" (UniqueName: \"kubernetes.io/projected/f4e81032-aa8b-4011-a300-24472e2b6ab1-kube-api-access-jf6lq\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:39.529551 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529536 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1934850b-46ad-4994-9166-22aa589e1a0d-bound-sa-token\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.529666 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4e81032-aa8b-4011-a300-24472e2b6ab1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:39.529666 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa3fecc-3119-4df7-84c5-0ffcaf0f9606-config-volume\") pod \"dns-default-zzmpt\" (UID: \"3aa3fecc-3119-4df7-84c5-0ffcaf0f9606\") " pod="openshift-dns/dns-default-zzmpt" Jul 16 21:10:39.529666 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9kcp\" (UniqueName: \"kubernetes.io/projected/1934850b-46ad-4994-9166-22aa589e1a0d-kube-api-access-j9kcp\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.529666 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529650 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa3fecc-3119-4df7-84c5-0ffcaf0f9606-tmp-dir\") pod \"dns-default-zzmpt\" (UID: \"3aa3fecc-3119-4df7-84c5-0ffcaf0f9606\") " pod="openshift-dns/dns-default-zzmpt" Jul 16 21:10:39.529845 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4c047a7-f534-4e17-ac3b-acb669c58df3-cert\") pod \"ingress-canary-dx929\" (UID: \"b4c047a7-f534-4e17-ac3b-acb669c58df3\") " pod="openshift-ingress-canary/ingress-canary-dx929" Jul 16 21:10:39.529845 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1934850b-46ad-4994-9166-22aa589e1a0d-ca-trust-extracted\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.529845 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1934850b-46ad-4994-9166-22aa589e1a0d-installation-pull-secrets\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.529845 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnb45\" (UniqueName: \"kubernetes.io/projected/3aa3fecc-3119-4df7-84c5-0ffcaf0f9606-kube-api-access-qnb45\") pod \"dns-default-zzmpt\" (UID: \"3aa3fecc-3119-4df7-84c5-0ffcaf0f9606\") " pod="openshift-dns/dns-default-zzmpt" Jul 16 21:10:39.529845 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529798 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f4e81032-aa8b-4011-a300-24472e2b6ab1-crio-socket\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:39.529845 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529815 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4e81032-aa8b-4011-a300-24472e2b6ab1-data-volume\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:39.529845 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529830 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1934850b-46ad-4994-9166-22aa589e1a0d-trusted-ca\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.529845 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529844 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3aa3fecc-3119-4df7-84c5-0ffcaf0f9606-metrics-tls\") pod \"dns-default-zzmpt\" (UID: \"3aa3fecc-3119-4df7-84c5-0ffcaf0f9606\") " pod="openshift-dns/dns-default-zzmpt" Jul 16 21:10:39.530122 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529944 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1934850b-46ad-4994-9166-22aa589e1a0d-registry-tls\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.530122 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.529983 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1934850b-46ad-4994-9166-22aa589e1a0d-registry-certificates\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.530122 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.530003 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t94k8\" (UniqueName: \"kubernetes.io/projected/b4c047a7-f534-4e17-ac3b-acb669c58df3-kube-api-access-t94k8\") pod \"ingress-canary-dx929\" (UID: \"b4c047a7-f534-4e17-ac3b-acb669c58df3\") " pod="openshift-ingress-canary/ingress-canary-dx929" Jul 16 21:10:39.531781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.531767 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 21:10:39.532091 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.532078 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 21:10:39.532151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.532090 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 21:10:39.532221 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.532090 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hmkvr\"" Jul 16 21:10:39.574760 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.574733 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dx929"] Jul 16 21:10:39.630358 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.630324 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4e81032-aa8b-4011-a300-24472e2b6ab1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:39.630358 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.630355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1934850b-46ad-4994-9166-22aa589e1a0d-image-registry-private-configuration\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.630571 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.630378 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf6lq\" (UniqueName: \"kubernetes.io/projected/f4e81032-aa8b-4011-a300-24472e2b6ab1-kube-api-access-jf6lq\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:39.630571 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.630401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1934850b-46ad-4994-9166-22aa589e1a0d-bound-sa-token\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.630571 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.630427 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4e81032-aa8b-4011-a300-24472e2b6ab1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:39.630571 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.630461 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa3fecc-3119-4df7-84c5-0ffcaf0f9606-config-volume\") pod \"dns-default-zzmpt\" (UID: \"3aa3fecc-3119-4df7-84c5-0ffcaf0f9606\") " pod="openshift-dns/dns-default-zzmpt" Jul 16 21:10:39.630571 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.630514 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9kcp\" (UniqueName: \"kubernetes.io/projected/1934850b-46ad-4994-9166-22aa589e1a0d-kube-api-access-j9kcp\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.630571 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.630562 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa3fecc-3119-4df7-84c5-0ffcaf0f9606-tmp-dir\") pod \"dns-default-zzmpt\" (UID: \"3aa3fecc-3119-4df7-84c5-0ffcaf0f9606\") " pod="openshift-dns/dns-default-zzmpt" Jul 16 21:10:39.631366 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.630596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4c047a7-f534-4e17-ac3b-acb669c58df3-cert\") pod \"ingress-canary-dx929\" (UID: \"b4c047a7-f534-4e17-ac3b-acb669c58df3\") " pod="openshift-ingress-canary/ingress-canary-dx929" Jul 16 21:10:39.631366 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.630629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1934850b-46ad-4994-9166-22aa589e1a0d-ca-trust-extracted\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.631366 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.630651 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1934850b-46ad-4994-9166-22aa589e1a0d-installation-pull-secrets\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.631366 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.630679 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnb45\" (UniqueName: \"kubernetes.io/projected/3aa3fecc-3119-4df7-84c5-0ffcaf0f9606-kube-api-access-qnb45\") pod \"dns-default-zzmpt\" (UID: \"3aa3fecc-3119-4df7-84c5-0ffcaf0f9606\") " pod="openshift-dns/dns-default-zzmpt" Jul 16 21:10:39.631366 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.630729 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f4e81032-aa8b-4011-a300-24472e2b6ab1-crio-socket\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:39.631366 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.630753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4e81032-aa8b-4011-a300-24472e2b6ab1-data-volume\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:39.631366 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.630798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1934850b-46ad-4994-9166-22aa589e1a0d-trusted-ca\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.631366 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.630822 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3aa3fecc-3119-4df7-84c5-0ffcaf0f9606-metrics-tls\") pod \"dns-default-zzmpt\" (UID: \"3aa3fecc-3119-4df7-84c5-0ffcaf0f9606\") " pod="openshift-dns/dns-default-zzmpt" Jul 16 21:10:39.631366 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.631102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1934850b-46ad-4994-9166-22aa589e1a0d-registry-tls\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.631366 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.631147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1934850b-46ad-4994-9166-22aa589e1a0d-registry-certificates\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.631366 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.631201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t94k8\" (UniqueName: \"kubernetes.io/projected/b4c047a7-f534-4e17-ac3b-acb669c58df3-kube-api-access-t94k8\") pod \"ingress-canary-dx929\" (UID: \"b4c047a7-f534-4e17-ac3b-acb669c58df3\") " pod="openshift-ingress-canary/ingress-canary-dx929" Jul 16 21:10:39.631880 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.631297 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f4e81032-aa8b-4011-a300-24472e2b6ab1-crio-socket\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:39.631880 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.631693 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa3fecc-3119-4df7-84c5-0ffcaf0f9606-tmp-dir\") pod \"dns-default-zzmpt\" (UID: \"3aa3fecc-3119-4df7-84c5-0ffcaf0f9606\") " pod="openshift-dns/dns-default-zzmpt" Jul 16 21:10:39.631880 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.631695 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1934850b-46ad-4994-9166-22aa589e1a0d-ca-trust-extracted\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.632344 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.632322 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4e81032-aa8b-4011-a300-24472e2b6ab1-data-volume\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:39.632742 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.632722 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1934850b-46ad-4994-9166-22aa589e1a0d-registry-certificates\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.633418 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.633400 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1934850b-46ad-4994-9166-22aa589e1a0d-trusted-ca\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.635088 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.635047 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1934850b-46ad-4994-9166-22aa589e1a0d-image-registry-private-configuration\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.635175 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.635108 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4c047a7-f534-4e17-ac3b-acb669c58df3-cert\") pod \"ingress-canary-dx929\" (UID: \"b4c047a7-f534-4e17-ac3b-acb669c58df3\") " pod="openshift-ingress-canary/ingress-canary-dx929" Jul 16 21:10:39.635175 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.635124 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1934850b-46ad-4994-9166-22aa589e1a0d-registry-tls\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.635175 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.635170 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1934850b-46ad-4994-9166-22aa589e1a0d-installation-pull-secrets\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.648782 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.648752 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnb45\" (UniqueName: \"kubernetes.io/projected/3aa3fecc-3119-4df7-84c5-0ffcaf0f9606-kube-api-access-qnb45\") pod \"dns-default-zzmpt\" (UID: \"3aa3fecc-3119-4df7-84c5-0ffcaf0f9606\") " pod="openshift-dns/dns-default-zzmpt" Jul 16 21:10:39.648991 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.648973 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t94k8\" (UniqueName: \"kubernetes.io/projected/b4c047a7-f534-4e17-ac3b-acb669c58df3-kube-api-access-t94k8\") pod \"ingress-canary-dx929\" (UID: \"b4c047a7-f534-4e17-ac3b-acb669c58df3\") " pod="openshift-ingress-canary/ingress-canary-dx929" Jul 16 21:10:39.649063 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.649033 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1934850b-46ad-4994-9166-22aa589e1a0d-bound-sa-token\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.662296 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.662278 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9kcp\" (UniqueName: \"kubernetes.io/projected/1934850b-46ad-4994-9166-22aa589e1a0d-kube-api-access-j9kcp\") pod \"image-registry-7cd4df479d-m2fwl\" (UID: \"1934850b-46ad-4994-9166-22aa589e1a0d\") " pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.759440 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.759367 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:39.837526 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.837284 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dx929" Jul 16 21:10:39.950380 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.950310 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cd4df479d-m2fwl"] Jul 16 21:10:39.955056 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:10:39.955027 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1934850b_46ad_4994_9166_22aa589e1a0d.slice/crio-4c17090e89a611b0fdc43f4365ac96ead5869870a4b0e69d5624e5089bdeb031 WatchSource:0}: Error finding container 4c17090e89a611b0fdc43f4365ac96ead5869870a4b0e69d5624e5089bdeb031: Status 404 returned error can't find the container with id 4c17090e89a611b0fdc43f4365ac96ead5869870a4b0e69d5624e5089bdeb031 Jul 16 21:10:39.983264 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:39.983241 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dx929"] Jul 16 21:10:39.985638 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:10:39.985617 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c047a7_f534_4e17_ac3b_acb669c58df3.slice/crio-bdad8b41d73ea924d6ee71468a04827101f9e755d10e8755ef3b485d22b8bf32 WatchSource:0}: Error finding container bdad8b41d73ea924d6ee71468a04827101f9e755d10e8755ef3b485d22b8bf32: Status 404 returned error can't find the container with id bdad8b41d73ea924d6ee71468a04827101f9e755d10e8755ef3b485d22b8bf32 Jul 16 21:10:40.342127 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:40.342047 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 21:10:40.351975 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:40.351944 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa3fecc-3119-4df7-84c5-0ffcaf0f9606-config-volume\") pod \"dns-default-zzmpt\" (UID: \"3aa3fecc-3119-4df7-84c5-0ffcaf0f9606\") " pod="openshift-dns/dns-default-zzmpt" Jul 16 21:10:40.620271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:40.620178 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 21:10:40.624621 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:40.624601 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3aa3fecc-3119-4df7-84c5-0ffcaf0f9606-metrics-tls\") pod \"dns-default-zzmpt\" (UID: \"3aa3fecc-3119-4df7-84c5-0ffcaf0f9606\") " pod="openshift-dns/dns-default-zzmpt" Jul 16 21:10:40.630534 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:40.630511 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: failed to sync configmap cache: timed out waiting for the condition Jul 16 21:10:40.630647 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:40.630597 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4e81032-aa8b-4011-a300-24472e2b6ab1-kube-rbac-proxy-cm podName:f4e81032-aa8b-4011-a300-24472e2b6ab1 nodeName:}" failed. No retries permitted until 2026-07-16 21:10:41.130577795 +0000 UTC m=+57.305685989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4e81032-aa8b-4011-a300-24472e2b6ab1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pjlc6" (UID: "f4e81032-aa8b-4011-a300-24472e2b6ab1") : failed to sync configmap cache: timed out waiting for the condition Jul 16 21:10:40.630714 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:40.630667 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: failed to sync secret cache: timed out waiting for the condition Jul 16 21:10:40.630763 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:40.630730 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4e81032-aa8b-4011-a300-24472e2b6ab1-insights-runtime-extractor-tls podName:f4e81032-aa8b-4011-a300-24472e2b6ab1 nodeName:}" failed. No retries permitted until 2026-07-16 21:10:41.130712422 +0000 UTC m=+57.305820610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4e81032-aa8b-4011-a300-24472e2b6ab1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pjlc6" (UID: "f4e81032-aa8b-4011-a300-24472e2b6ab1") : failed to sync secret cache: timed out waiting for the condition Jul 16 21:10:40.644623 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:40.644562 2577 projected.go:291] Couldn't get configMap openshift-insights/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Jul 16 21:10:40.661969 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:40.661945 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 21:10:40.663804 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:40.663780 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 21:10:40.665524 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:40.665490 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dx929" event={"ID":"b4c047a7-f534-4e17-ac3b-acb669c58df3","Type":"ContainerStarted","Data":"bdad8b41d73ea924d6ee71468a04827101f9e755d10e8755ef3b485d22b8bf32"} Jul 16 21:10:40.667093 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:40.667070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" event={"ID":"1934850b-46ad-4994-9166-22aa589e1a0d","Type":"ContainerStarted","Data":"7f870bdbc816f8f558ca8340cb5c7a576922701703e79f3bdbe158facf11dc1f"} Jul 16 21:10:40.667206 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:40.667110 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" event={"ID":"1934850b-46ad-4994-9166-22aa589e1a0d","Type":"ContainerStarted","Data":"4c17090e89a611b0fdc43f4365ac96ead5869870a4b0e69d5624e5089bdeb031"} Jul 16 21:10:40.667270 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:40.667220 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:10:40.694661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:40.694620 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" podStartSLOduration=1.694605924 podStartE2EDuration="1.694605924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:40.693766157 +0000 UTC m=+56.868874365" watchObservedRunningTime="2026-07-16 21:10:40.694605924 +0000 UTC m=+56.869714115" Jul 16 21:10:40.696588 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:40.696567 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 21:10:40.714993 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:40.714966 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-glgkx\"" Jul 16 21:10:40.881361 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:40.881340 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5s8t4\"" Jul 16 21:10:40.888817 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:40.888798 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zzmpt" Jul 16 21:10:41.038914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:41.038881 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zzmpt"] Jul 16 21:10:41.042557 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:10:41.042528 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa3fecc_3119_4df7_84c5_0ffcaf0f9606.slice/crio-f5aa0b1980a2a72c5d8a096644e62c6a79994aa66a233492a5e17e73033cb022 WatchSource:0}: Error finding container f5aa0b1980a2a72c5d8a096644e62c6a79994aa66a233492a5e17e73033cb022: Status 404 returned error can't find the container with id f5aa0b1980a2a72c5d8a096644e62c6a79994aa66a233492a5e17e73033cb022 Jul 16 21:10:41.091079 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:41.091054 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 21:10:41.095147 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:41.095120 2577 projected.go:196] Error preparing data for projected volume kube-api-access-jf6lq for pod openshift-insights/insights-runtime-extractor-pjlc6: failed to sync configmap cache: timed out waiting for the condition Jul 16 21:10:41.095287 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:41.095218 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4e81032-aa8b-4011-a300-24472e2b6ab1-kube-api-access-jf6lq podName:f4e81032-aa8b-4011-a300-24472e2b6ab1 nodeName:}" failed. No retries permitted until 2026-07-16 21:10:41.595197445 +0000 UTC m=+57.770305636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jf6lq" (UniqueName: "kubernetes.io/projected/f4e81032-aa8b-4011-a300-24472e2b6ab1-kube-api-access-jf6lq") pod "insights-runtime-extractor-pjlc6" (UID: "f4e81032-aa8b-4011-a300-24472e2b6ab1") : failed to sync configmap cache: timed out waiting for the condition Jul 16 21:10:41.145981 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:41.145903 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4e81032-aa8b-4011-a300-24472e2b6ab1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:41.146101 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:41.146035 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4e81032-aa8b-4011-a300-24472e2b6ab1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:41.150146 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:41.147098 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4e81032-aa8b-4011-a300-24472e2b6ab1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:41.150913 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:41.150889 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4e81032-aa8b-4011-a300-24472e2b6ab1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:41.648832 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:41.648790 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf6lq\" (UniqueName: \"kubernetes.io/projected/f4e81032-aa8b-4011-a300-24472e2b6ab1-kube-api-access-jf6lq\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:41.651936 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:41.651908 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf6lq\" (UniqueName: \"kubernetes.io/projected/f4e81032-aa8b-4011-a300-24472e2b6ab1-kube-api-access-jf6lq\") pod \"insights-runtime-extractor-pjlc6\" (UID: \"f4e81032-aa8b-4011-a300-24472e2b6ab1\") " pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:41.671240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:41.671207 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zzmpt" event={"ID":"3aa3fecc-3119-4df7-84c5-0ffcaf0f9606","Type":"ContainerStarted","Data":"f5aa0b1980a2a72c5d8a096644e62c6a79994aa66a233492a5e17e73033cb022"} Jul 16 21:10:41.891602 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:41.891566 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pjlc6" Jul 16 21:10:42.856467 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:42.856439 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-flcfc"] Jul 16 21:10:42.858337 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:42.858318 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-flcfc" Jul 16 21:10:42.862537 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:42.862513 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 16 21:10:42.862642 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:42.862521 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-fw7ps\"" Jul 16 21:10:42.876108 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:42.876088 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-flcfc"] Jul 16 21:10:42.955958 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:42.955925 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/92605663-a637-4721-b2a9-c37ea6873763-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-flcfc\" (UID: \"92605663-a637-4721-b2a9-c37ea6873763\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-flcfc" Jul 16 21:10:43.034576 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:43.034546 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pjlc6"] Jul 16 21:10:43.037633 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:10:43.037611 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e81032_aa8b_4011_a300_24472e2b6ab1.slice/crio-a69cf38787bcbea695625d0531b4fd419ed50ee35eae6fc47a0d786e316c15ee WatchSource:0}: Error finding container a69cf38787bcbea695625d0531b4fd419ed50ee35eae6fc47a0d786e316c15ee: Status 404 returned error can't find the container with id a69cf38787bcbea695625d0531b4fd419ed50ee35eae6fc47a0d786e316c15ee Jul 16 21:10:43.056525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:43.056498 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/92605663-a637-4721-b2a9-c37ea6873763-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-flcfc\" (UID: \"92605663-a637-4721-b2a9-c37ea6873763\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-flcfc" Jul 16 21:10:43.058859 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:43.058813 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/92605663-a637-4721-b2a9-c37ea6873763-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-flcfc\" (UID: \"92605663-a637-4721-b2a9-c37ea6873763\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-flcfc" Jul 16 21:10:43.166746 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:43.166716 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-flcfc" Jul 16 21:10:43.294177 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:43.294142 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-flcfc"] Jul 16 21:10:43.297886 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:10:43.297862 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92605663_a637_4721_b2a9_c37ea6873763.slice/crio-0b5b811f2ee8f2fe0cba50b96e3e69ddc7f772a7036f9ee1abc445319f830209 WatchSource:0}: Error finding container 0b5b811f2ee8f2fe0cba50b96e3e69ddc7f772a7036f9ee1abc445319f830209: Status 404 returned error can't find the container with id 0b5b811f2ee8f2fe0cba50b96e3e69ddc7f772a7036f9ee1abc445319f830209 Jul 16 21:10:43.679533 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:43.679495 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dx929" event={"ID":"b4c047a7-f534-4e17-ac3b-acb669c58df3","Type":"ContainerStarted","Data":"6fb51d1d4e0a5ac1cf68fa0ba99b7a8e1a40db7430d7ef3a4d724906b7bec011"} Jul 16 21:10:43.681755 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:43.681695 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zzmpt" event={"ID":"3aa3fecc-3119-4df7-84c5-0ffcaf0f9606","Type":"ContainerStarted","Data":"ce10e5cfc1c4d93361d2c70b74c2b84c72e1f43ae95ff4a36d8135029310bb68"} Jul 16 21:10:43.681877 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:43.681784 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zzmpt" event={"ID":"3aa3fecc-3119-4df7-84c5-0ffcaf0f9606","Type":"ContainerStarted","Data":"b75de90b2dc86ede76f05b2ce9fc37cbe7ba8d2f6a791b70693aa8b3f4d9deb2"} Jul 16 21:10:43.681877 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:43.681814 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zzmpt" Jul 16 21:10:43.683861 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:43.683834 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pjlc6" event={"ID":"f4e81032-aa8b-4011-a300-24472e2b6ab1","Type":"ContainerStarted","Data":"a69cf38787bcbea695625d0531b4fd419ed50ee35eae6fc47a0d786e316c15ee"} Jul 16 21:10:43.685309 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:43.685284 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-flcfc" event={"ID":"92605663-a637-4721-b2a9-c37ea6873763","Type":"ContainerStarted","Data":"0b5b811f2ee8f2fe0cba50b96e3e69ddc7f772a7036f9ee1abc445319f830209"} Jul 16 21:10:43.765357 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:43.765298 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zzmpt" podStartSLOduration=2.922030652 podStartE2EDuration="4.765275571s" totalImagesPullingTime="1.843244919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:43.765133441 +0000 UTC m=+59.940241634" watchObservedRunningTime="2026-07-16 21:10:43.765275571 +0000 UTC m=+59.940383766" Jul 16 21:10:43.766159 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:43.766120 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dx929" podStartSLOduration=1.8661301510000001 podStartE2EDuration="4.766108054s" totalImagesPullingTime="2.899977903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:43.71583097 +0000 UTC m=+59.890939187" watchObservedRunningTime="2026-07-16 21:10:43.766108054 +0000 UTC m=+59.941216248" Jul 16 21:10:44.605751 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:44.605716 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5lc2" Jul 16 21:10:44.690849 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:44.690791 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-flcfc" event={"ID":"92605663-a637-4721-b2a9-c37ea6873763","Type":"ContainerStarted","Data":"30e78875de7b00372f608c7371fb755c0c9b33d047f8431422bdf8e725dc3978"} Jul 16 21:10:44.730376 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:44.730317 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-flcfc" podStartSLOduration=1.539483845 podStartE2EDuration="2.730294639s" totalImagesPullingTime="1.190810794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:44.72862477 +0000 UTC m=+60.903732963" watchObservedRunningTime="2026-07-16 21:10:44.730294639 +0000 UTC m=+60.905402831" Jul 16 21:10:45.698211 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:45.696442 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-flcfc" Jul 16 21:10:45.701762 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:45.701730 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-flcfc" Jul 16 21:10:48.707320 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:48.707276 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pjlc6" event={"ID":"f4e81032-aa8b-4011-a300-24472e2b6ab1","Type":"ContainerStarted","Data":"7164af131c023a09a3be730cc1cb336592c3cd11c814959b3178d8c9c30ec643"} Jul 16 21:10:50.013309 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:50.013268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret\") pod \"global-pull-secret-syncer-2xbp6\" (UID: \"3bd19c30-8019-4896-aa14-62214c3e042c\") " pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:10:50.013753 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:50.013340 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs\") pod \"network-metrics-daemon-twm75\" (UID: \"9a4e202f-99dc-4068-9496-acb28f0552b5\") " pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:10:50.016588 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:50.016559 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 21:10:50.017684 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:50.017668 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 21:10:50.026423 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:50.026398 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4e202f-99dc-4068-9496-acb28f0552b5-metrics-certs\") pod \"network-metrics-daemon-twm75\" (UID: \"9a4e202f-99dc-4068-9496-acb28f0552b5\") " pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:10:50.026549 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:50.026400 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd19c30-8019-4896-aa14-62214c3e042c-original-pull-secret\") pod \"global-pull-secret-syncer-2xbp6\" (UID: \"3bd19c30-8019-4896-aa14-62214c3e042c\") " pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:10:50.059232 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:50.059204 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ljktf\"" Jul 16 21:10:50.065851 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:50.065824 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twm75" Jul 16 21:10:50.164788 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:50.164753 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xbp6" Jul 16 21:10:50.214819 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:50.214789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzsk\" (UniqueName: \"kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk\") pod \"network-check-target-dbknf\" (UID: \"d278255e-2c35-4471-bbd9-095d9c476896\") " pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:50.220385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:50.220360 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 21:10:50.227834 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:50.227813 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 21:10:50.238060 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:50.238034 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vzsk\" (UniqueName: \"kubernetes.io/projected/d278255e-2c35-4471-bbd9-095d9c476896-kube-api-access-5vzsk\") pod \"network-check-target-dbknf\" (UID: \"d278255e-2c35-4471-bbd9-095d9c476896\") " pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:50.463265 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:50.463233 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xwmds\"" Jul 16 21:10:50.470154 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:50.470130 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:51.415621 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:51.415540 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2xbp6"] Jul 16 21:10:51.418357 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:10:51.418318 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd19c30_8019_4896_aa14_62214c3e042c.slice/crio-da089a8a8b92e2b0e8bee3bccc6fdfe33d5366a33dabdacb1e29263313eed09e WatchSource:0}: Error finding container da089a8a8b92e2b0e8bee3bccc6fdfe33d5366a33dabdacb1e29263313eed09e: Status 404 returned error can't find the container with id da089a8a8b92e2b0e8bee3bccc6fdfe33d5366a33dabdacb1e29263313eed09e Jul 16 21:10:51.439775 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:51.439756 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dbknf"] Jul 16 21:10:51.440557 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:10:51.440535 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd278255e_2c35_4471_bbd9_095d9c476896.slice/crio-bc2d3459a8521e355b210d93c5e99ed91ccb1364eafe17d04f760d459c3d7429 WatchSource:0}: Error finding container bc2d3459a8521e355b210d93c5e99ed91ccb1364eafe17d04f760d459c3d7429: Status 404 returned error can't find the container with id bc2d3459a8521e355b210d93c5e99ed91ccb1364eafe17d04f760d459c3d7429 Jul 16 21:10:51.444033 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:51.444011 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-twm75"] Jul 16 21:10:51.446986 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:10:51.446965 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4e202f_99dc_4068_9496_acb28f0552b5.slice/crio-7da258d50cf19122321051573e92517b696ab3c24ec9e17cc5946c0c828c172b WatchSource:0}: Error finding container 7da258d50cf19122321051573e92517b696ab3c24ec9e17cc5946c0c828c172b: Status 404 returned error can't find the container with id 7da258d50cf19122321051573e92517b696ab3c24ec9e17cc5946c0c828c172b Jul 16 21:10:51.719657 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:51.719620 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pjlc6" event={"ID":"f4e81032-aa8b-4011-a300-24472e2b6ab1","Type":"ContainerStarted","Data":"141a5f17aafa040206eb6c961a615c189c1f9627d71ce8031f8e80139db1f1fe"} Jul 16 21:10:51.720770 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:51.720736 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2xbp6" event={"ID":"3bd19c30-8019-4896-aa14-62214c3e042c","Type":"ContainerStarted","Data":"da089a8a8b92e2b0e8bee3bccc6fdfe33d5366a33dabdacb1e29263313eed09e"} Jul 16 21:10:51.722025 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:51.721998 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-twm75" event={"ID":"9a4e202f-99dc-4068-9496-acb28f0552b5","Type":"ContainerStarted","Data":"7da258d50cf19122321051573e92517b696ab3c24ec9e17cc5946c0c828c172b"} Jul 16 21:10:51.723155 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:51.723121 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dbknf" event={"ID":"d278255e-2c35-4471-bbd9-095d9c476896","Type":"ContainerStarted","Data":"bc2d3459a8521e355b210d93c5e99ed91ccb1364eafe17d04f760d459c3d7429"} Jul 16 21:10:52.841772 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.841644 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-mqs5r"] Jul 16 21:10:52.859083 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.858652 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n"] Jul 16 21:10:52.859083 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.859050 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:52.871957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.869435 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 16 21:10:52.871957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.869693 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wz8qn\"" Jul 16 21:10:52.871957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.869939 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 16 21:10:52.871957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.869892 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 21:10:52.871957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.870123 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 21:10:52.871957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.870228 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 21:10:52.871957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.871944 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-mqs5r"] Jul 16 21:10:52.872345 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.871990 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-klcvk"] Jul 16 21:10:52.873049 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.872490 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" Jul 16 21:10:52.873431 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.873415 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 16 21:10:52.880176 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.880095 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 16 21:10:52.880808 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.880618 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 16 21:10:52.881108 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.880934 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2n4sm\"" Jul 16 21:10:52.888581 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.887578 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n"] Jul 16 21:10:52.888581 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.887705 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:52.890653 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.890409 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5vfpq\"" Jul 16 21:10:52.890744 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.890674 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 21:10:52.891125 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.891108 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 21:10:52.892098 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.891916 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 21:10:52.936631 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.936589 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-accelerators-collector-config\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:52.936769 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.936646 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/764206ea-3f09-45a2-aae6-47dd7908d230-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:52.936769 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.936687 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcxxb\" (UniqueName: \"kubernetes.io/projected/cf548327-8c0f-4961-8c7b-d547ff8a9426-kube-api-access-fcxxb\") pod \"openshift-state-metrics-79f6958cdf-c5b5n\" (UID: \"cf548327-8c0f-4961-8c7b-d547ff8a9426\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" Jul 16 21:10:52.936769 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.936720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/764206ea-3f09-45a2-aae6-47dd7908d230-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:52.936769 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.936758 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf548327-8c0f-4961-8c7b-d547ff8a9426-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-c5b5n\" (UID: \"cf548327-8c0f-4961-8c7b-d547ff8a9426\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" Jul 16 21:10:52.936986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.936785 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-tls\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:52.936986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.936815 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/764206ea-3f09-45a2-aae6-47dd7908d230-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:52.936986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.936839 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a36431c2-8598-4a45-b16c-11ac06c52c42-metrics-client-ca\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:52.936986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.936866 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:52.936986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.936894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46mwf\" (UniqueName: \"kubernetes.io/projected/a36431c2-8598-4a45-b16c-11ac06c52c42-kube-api-access-46mwf\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:52.936986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.936922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-textfile\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:52.936986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.936947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-wtmp\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:52.936986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.936975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/764206ea-3f09-45a2-aae6-47dd7908d230-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:52.937383 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.937003 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf548327-8c0f-4961-8c7b-d547ff8a9426-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-c5b5n\" (UID: \"cf548327-8c0f-4961-8c7b-d547ff8a9426\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" Jul 16 21:10:52.937383 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.937029 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a36431c2-8598-4a45-b16c-11ac06c52c42-root\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:52.937383 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.937088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/764206ea-3f09-45a2-aae6-47dd7908d230-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:52.937383 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.937211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a36431c2-8598-4a45-b16c-11ac06c52c42-sys\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:52.937383 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.937291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf548327-8c0f-4961-8c7b-d547ff8a9426-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-c5b5n\" (UID: \"cf548327-8c0f-4961-8c7b-d547ff8a9426\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" Jul 16 21:10:52.937383 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:52.937380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g7xg\" (UniqueName: \"kubernetes.io/projected/764206ea-3f09-45a2-aae6-47dd7908d230-kube-api-access-4g7xg\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:53.038501 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038464 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a36431c2-8598-4a45-b16c-11ac06c52c42-sys\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.038660 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf548327-8c0f-4961-8c7b-d547ff8a9426-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-c5b5n\" (UID: \"cf548327-8c0f-4961-8c7b-d547ff8a9426\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" Jul 16 21:10:53.038660 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038541 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g7xg\" (UniqueName: \"kubernetes.io/projected/764206ea-3f09-45a2-aae6-47dd7908d230-kube-api-access-4g7xg\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:53.038660 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-accelerators-collector-config\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.038660 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/764206ea-3f09-45a2-aae6-47dd7908d230-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:53.038660 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038603 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a36431c2-8598-4a45-b16c-11ac06c52c42-sys\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.038660 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcxxb\" (UniqueName: \"kubernetes.io/projected/cf548327-8c0f-4961-8c7b-d547ff8a9426-kube-api-access-fcxxb\") pod \"openshift-state-metrics-79f6958cdf-c5b5n\" (UID: \"cf548327-8c0f-4961-8c7b-d547ff8a9426\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" Jul 16 21:10:53.038660 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/764206ea-3f09-45a2-aae6-47dd7908d230-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:53.039028 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf548327-8c0f-4961-8c7b-d547ff8a9426-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-c5b5n\" (UID: \"cf548327-8c0f-4961-8c7b-d547ff8a9426\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" Jul 16 21:10:53.039028 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-tls\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.039028 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038729 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/764206ea-3f09-45a2-aae6-47dd7908d230-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:53.039028 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a36431c2-8598-4a45-b16c-11ac06c52c42-metrics-client-ca\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.039028 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.039028 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46mwf\" (UniqueName: \"kubernetes.io/projected/a36431c2-8598-4a45-b16c-11ac06c52c42-kube-api-access-46mwf\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.039028 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-textfile\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.039028 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-wtmp\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.039028 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/764206ea-3f09-45a2-aae6-47dd7908d230-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:53.039028 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038939 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf548327-8c0f-4961-8c7b-d547ff8a9426-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-c5b5n\" (UID: \"cf548327-8c0f-4961-8c7b-d547ff8a9426\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" Jul 16 21:10:53.039028 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.038973 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a36431c2-8598-4a45-b16c-11ac06c52c42-root\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.039028 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.039012 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/764206ea-3f09-45a2-aae6-47dd7908d230-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:53.039600 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.039246 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-accelerators-collector-config\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.039654 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.039602 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-wtmp\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.039710 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.039697 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a36431c2-8598-4a45-b16c-11ac06c52c42-metrics-client-ca\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.040240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.040214 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/764206ea-3f09-45a2-aae6-47dd7908d230-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:53.040321 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.040285 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a36431c2-8598-4a45-b16c-11ac06c52c42-root\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.040321 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:53.040316 2577 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 16 21:10:53.040430 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:53.040375 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/764206ea-3f09-45a2-aae6-47dd7908d230-kube-state-metrics-tls podName:764206ea-3f09-45a2-aae6-47dd7908d230 nodeName:}" failed. No retries permitted until 2026-07-16 21:10:53.540354475 +0000 UTC m=+69.715462662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/764206ea-3f09-45a2-aae6-47dd7908d230-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-mqs5r" (UID: "764206ea-3f09-45a2-aae6-47dd7908d230") : secret "kube-state-metrics-tls" not found Jul 16 21:10:53.040732 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:53.040705 2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 16 21:10:53.040823 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:53.040754 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-tls podName:a36431c2-8598-4a45-b16c-11ac06c52c42 nodeName:}" failed. No retries permitted until 2026-07-16 21:10:53.540740341 +0000 UTC m=+69.715848529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-tls") pod "node-exporter-klcvk" (UID: "a36431c2-8598-4a45-b16c-11ac06c52c42") : secret "node-exporter-tls" not found Jul 16 21:10:53.040882 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.040864 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/764206ea-3f09-45a2-aae6-47dd7908d230-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:53.041114 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.041093 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-textfile\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.041402 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.041378 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf548327-8c0f-4961-8c7b-d547ff8a9426-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-c5b5n\" (UID: \"cf548327-8c0f-4961-8c7b-d547ff8a9426\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" Jul 16 21:10:53.041832 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.041810 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/764206ea-3f09-45a2-aae6-47dd7908d230-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:53.045398 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.045373 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf548327-8c0f-4961-8c7b-d547ff8a9426-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-c5b5n\" (UID: \"cf548327-8c0f-4961-8c7b-d547ff8a9426\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" Jul 16 21:10:53.045507 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.045487 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf548327-8c0f-4961-8c7b-d547ff8a9426-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-c5b5n\" (UID: \"cf548327-8c0f-4961-8c7b-d547ff8a9426\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" Jul 16 21:10:53.046942 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.046896 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.047545 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.047520 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/764206ea-3f09-45a2-aae6-47dd7908d230-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:53.061652 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.061605 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46mwf\" (UniqueName: \"kubernetes.io/projected/a36431c2-8598-4a45-b16c-11ac06c52c42-kube-api-access-46mwf\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.061773 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.061754 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g7xg\" (UniqueName: \"kubernetes.io/projected/764206ea-3f09-45a2-aae6-47dd7908d230-kube-api-access-4g7xg\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:53.072672 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.072630 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcxxb\" (UniqueName: \"kubernetes.io/projected/cf548327-8c0f-4961-8c7b-d547ff8a9426-kube-api-access-fcxxb\") pod \"openshift-state-metrics-79f6958cdf-c5b5n\" (UID: \"cf548327-8c0f-4961-8c7b-d547ff8a9426\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" Jul 16 21:10:53.188499 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.188467 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" Jul 16 21:10:53.527532 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.527231 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n"] Jul 16 21:10:53.529942 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:10:53.529913 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf548327_8c0f_4961_8c7b_d547ff8a9426.slice/crio-d6f147c2f0aaea984c4b3dfd41ae33e4e31d249de0061dd52eb4822a776a7ea6 WatchSource:0}: Error finding container d6f147c2f0aaea984c4b3dfd41ae33e4e31d249de0061dd52eb4822a776a7ea6: Status 404 returned error can't find the container with id d6f147c2f0aaea984c4b3dfd41ae33e4e31d249de0061dd52eb4822a776a7ea6 Jul 16 21:10:53.543200 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.543151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-tls\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.543309 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.543267 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/764206ea-3f09-45a2-aae6-47dd7908d230-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:53.545787 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.545764 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/764206ea-3f09-45a2-aae6-47dd7908d230-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-mqs5r\" (UID: \"764206ea-3f09-45a2-aae6-47dd7908d230\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:53.547611 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.547566 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a36431c2-8598-4a45-b16c-11ac06c52c42-node-exporter-tls\") pod \"node-exporter-klcvk\" (UID: \"a36431c2-8598-4a45-b16c-11ac06c52c42\") " pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.693480 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.693450 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zzmpt" Jul 16 21:10:53.736172 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.736091 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" event={"ID":"cf548327-8c0f-4961-8c7b-d547ff8a9426","Type":"ContainerStarted","Data":"d6f147c2f0aaea984c4b3dfd41ae33e4e31d249de0061dd52eb4822a776a7ea6"} Jul 16 21:10:53.738388 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.738357 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-twm75" event={"ID":"9a4e202f-99dc-4068-9496-acb28f0552b5","Type":"ContainerStarted","Data":"071d62f54bbf4d3900a08053e8c1b4f3b0f647bd95a3b29b3aa1d551919f5ea9"} Jul 16 21:10:53.773005 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.772781 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" Jul 16 21:10:53.800387 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.800353 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-klcvk" Jul 16 21:10:53.812946 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.812919 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:10:53.831606 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.831582 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:53.846116 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.845900 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 21:10:53.846116 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.846016 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 21:10:53.847081 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.846804 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 21:10:53.847081 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.846841 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8d64h\"" Jul 16 21:10:53.847300 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.847283 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 21:10:53.847479 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.847458 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 21:10:53.847818 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.847798 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:10:53.848162 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.848037 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 21:10:53.848162 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.848080 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 21:10:53.848708 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.848457 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 21:10:53.854175 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.854150 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 21:10:53.947109 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.947075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fb17380-24e0-4749-93ed-48ce0e0dafb2-config-out\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:53.947306 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.947124 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:53.947306 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.947226 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-config-volume\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:53.947306 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.947261 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:53.947306 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.947296 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fb17380-24e0-4749-93ed-48ce0e0dafb2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:53.947514 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.947361 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fb17380-24e0-4749-93ed-48ce0e0dafb2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:53.947514 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.947389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:53.947514 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.947425 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:53.947514 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.947450 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:53.947514 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.947481 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9fb17380-24e0-4749-93ed-48ce0e0dafb2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:53.947514 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.947508 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fb17380-24e0-4749-93ed-48ce0e0dafb2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:53.947849 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.947539 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-web-config\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:53.947849 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:53.947564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8659\" (UniqueName: \"kubernetes.io/projected/9fb17380-24e0-4749-93ed-48ce0e0dafb2-kube-api-access-f8659\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.048813 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.048720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fb17380-24e0-4749-93ed-48ce0e0dafb2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.048813 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.048779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fb17380-24e0-4749-93ed-48ce0e0dafb2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.048813 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.048811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.049099 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.048844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.049099 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.048879 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.049099 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.048915 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9fb17380-24e0-4749-93ed-48ce0e0dafb2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.049099 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.048944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fb17380-24e0-4749-93ed-48ce0e0dafb2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.049099 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.048969 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-web-config\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.049099 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.049000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8659\" (UniqueName: \"kubernetes.io/projected/9fb17380-24e0-4749-93ed-48ce0e0dafb2-kube-api-access-f8659\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.049099 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.049046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fb17380-24e0-4749-93ed-48ce0e0dafb2-config-out\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.049099 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.049077 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.049526 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.049113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-config-volume\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.049526 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.049137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.049802 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.049778 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fb17380-24e0-4749-93ed-48ce0e0dafb2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.049890 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.049858 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fb17380-24e0-4749-93ed-48ce0e0dafb2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.050483 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.050459 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9fb17380-24e0-4749-93ed-48ce0e0dafb2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.052878 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.052853 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-config-volume\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.053602 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.053576 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.053828 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.053783 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fb17380-24e0-4749-93ed-48ce0e0dafb2-config-out\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.053932 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.053837 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.054453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.054407 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-web-config\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.054582 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.054506 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.054647 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.054622 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.055086 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.055059 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fb17380-24e0-4749-93ed-48ce0e0dafb2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.055905 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.055882 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.062415 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.062374 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8659\" (UniqueName: \"kubernetes.io/projected/9fb17380-24e0-4749-93ed-48ce0e0dafb2-kube-api-access-f8659\") pod \"alertmanager-main-0\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:54.145467 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:54.145421 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:10:55.581907 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.581872 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f5cff5975-jsgmr"] Jul 16 21:10:55.586856 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.586826 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.590405 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.590378 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 21:10:55.591202 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.591165 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 21:10:55.591202 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.591175 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 21:10:55.591666 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.591648 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 21:10:55.591768 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.591685 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 21:10:55.592553 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.592537 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9nxmq\"" Jul 16 21:10:55.592864 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.592826 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 21:10:55.592864 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.592845 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 21:10:55.596418 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.596399 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 21:10:55.601889 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.601872 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f5cff5975-jsgmr"] Jul 16 21:10:55.660537 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.660509 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-trusted-ca-bundle\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.660702 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.660546 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-oauth-serving-cert\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.660702 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.660568 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-serving-cert\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.660702 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.660600 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-oauth-config\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.660702 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.660666 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-service-ca\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.660908 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.660718 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-config\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.660908 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.660750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m67wg\" (UniqueName: \"kubernetes.io/projected/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-kube-api-access-m67wg\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.761751 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.761673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-trusted-ca-bundle\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.761751 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.761718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-oauth-serving-cert\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.761751 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.761753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-serving-cert\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.762031 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.761781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-oauth-config\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.762031 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.761805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-service-ca\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.762031 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.761838 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-config\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.762031 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.761865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m67wg\" (UniqueName: \"kubernetes.io/projected/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-kube-api-access-m67wg\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.762720 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.762666 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-oauth-serving-cert\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.762720 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.762686 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-trusted-ca-bundle\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.762887 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.762760 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-service-ca\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.762887 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.762760 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-config\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.764866 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.764846 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-serving-cert\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.764975 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.764942 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-oauth-config\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.775762 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.775742 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m67wg\" (UniqueName: \"kubernetes.io/projected/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-kube-api-access-m67wg\") pod \"console-f5cff5975-jsgmr\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:55.898481 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:55.898442 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:10:57.201224 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.201172 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg"] Jul 16 21:10:57.205119 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.205034 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg" Jul 16 21:10:57.208397 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.208371 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-fhpj6\"" Jul 16 21:10:57.208992 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.208959 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 16 21:10:57.211957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.211936 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg"] Jul 16 21:10:57.262027 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.261994 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7479cf6c97-59v8t"] Jul 16 21:10:57.264453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.264433 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.267223 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.267157 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 21:10:57.267340 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.267242 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 16 21:10:57.268448 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.268430 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 16 21:10:57.268562 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.268463 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-78jv4lookaag0\"" Jul 16 21:10:57.268562 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.268466 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 16 21:10:57.268562 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.268481 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-p6z7w\"" Jul 16 21:10:57.273793 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.273768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e4bb1fec-1520-4f19-ba3d-c95eb209e8dd-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-gddmg\" (UID: \"e4bb1fec-1520-4f19-ba3d-c95eb209e8dd\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg" Jul 16 21:10:57.273902 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.273806 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-564tr\" (UniqueName: \"kubernetes.io/projected/e4bb1fec-1520-4f19-ba3d-c95eb209e8dd-kube-api-access-564tr\") pod \"monitoring-plugin-7b7b7bc8bf-gddmg\" (UID: \"e4bb1fec-1520-4f19-ba3d-c95eb209e8dd\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg" Jul 16 21:10:57.277384 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.277364 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7479cf6c97-59v8t"] Jul 16 21:10:57.374562 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.374519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/202b3f86-d79e-457b-abd8-9fe67f18fb69-client-ca-bundle\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.374728 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.374573 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e4bb1fec-1520-4f19-ba3d-c95eb209e8dd-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-gddmg\" (UID: \"e4bb1fec-1520-4f19-ba3d-c95eb209e8dd\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg" Jul 16 21:10:57.374728 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.374597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-564tr\" (UniqueName: \"kubernetes.io/projected/e4bb1fec-1520-4f19-ba3d-c95eb209e8dd-kube-api-access-564tr\") pod \"monitoring-plugin-7b7b7bc8bf-gddmg\" (UID: \"e4bb1fec-1520-4f19-ba3d-c95eb209e8dd\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg" Jul 16 21:10:57.374728 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.374617 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/202b3f86-d79e-457b-abd8-9fe67f18fb69-metrics-server-audit-profiles\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.374728 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.374641 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/202b3f86-d79e-457b-abd8-9fe67f18fb69-secret-metrics-server-tls\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.374728 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.374674 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/202b3f86-d79e-457b-abd8-9fe67f18fb69-audit-log\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.374728 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.374698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtx9t\" (UniqueName: \"kubernetes.io/projected/202b3f86-d79e-457b-abd8-9fe67f18fb69-kube-api-access-dtx9t\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.375003 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.374740 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/202b3f86-d79e-457b-abd8-9fe67f18fb69-secret-metrics-server-client-certs\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.375003 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:57.374743 2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 16 21:10:57.375003 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.374768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/202b3f86-d79e-457b-abd8-9fe67f18fb69-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.375003 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:10:57.374817 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4bb1fec-1520-4f19-ba3d-c95eb209e8dd-monitoring-plugin-cert podName:e4bb1fec-1520-4f19-ba3d-c95eb209e8dd nodeName:}" failed. No retries permitted until 2026-07-16 21:10:57.874795035 +0000 UTC m=+74.049903219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/e4bb1fec-1520-4f19-ba3d-c95eb209e8dd-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-gddmg" (UID: "e4bb1fec-1520-4f19-ba3d-c95eb209e8dd") : secret "monitoring-plugin-cert" not found Jul 16 21:10:57.388379 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.388348 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-564tr\" (UniqueName: \"kubernetes.io/projected/e4bb1fec-1520-4f19-ba3d-c95eb209e8dd-kube-api-access-564tr\") pod \"monitoring-plugin-7b7b7bc8bf-gddmg\" (UID: \"e4bb1fec-1520-4f19-ba3d-c95eb209e8dd\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg" Jul 16 21:10:57.475602 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.475518 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/202b3f86-d79e-457b-abd8-9fe67f18fb69-metrics-server-audit-profiles\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.475755 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.475643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/202b3f86-d79e-457b-abd8-9fe67f18fb69-secret-metrics-server-tls\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.475755 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.475702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/202b3f86-d79e-457b-abd8-9fe67f18fb69-audit-log\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.475755 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.475736 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtx9t\" (UniqueName: \"kubernetes.io/projected/202b3f86-d79e-457b-abd8-9fe67f18fb69-kube-api-access-dtx9t\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.475915 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.475788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/202b3f86-d79e-457b-abd8-9fe67f18fb69-secret-metrics-server-client-certs\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.475915 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.475827 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/202b3f86-d79e-457b-abd8-9fe67f18fb69-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.475915 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.475893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/202b3f86-d79e-457b-abd8-9fe67f18fb69-client-ca-bundle\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.476558 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.476482 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/202b3f86-d79e-457b-abd8-9fe67f18fb69-audit-log\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.476679 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.476661 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/202b3f86-d79e-457b-abd8-9fe67f18fb69-metrics-server-audit-profiles\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.476903 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.476883 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/202b3f86-d79e-457b-abd8-9fe67f18fb69-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.478766 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.478741 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/202b3f86-d79e-457b-abd8-9fe67f18fb69-client-ca-bundle\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.478887 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.478852 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/202b3f86-d79e-457b-abd8-9fe67f18fb69-secret-metrics-server-client-certs\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.478987 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.478932 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/202b3f86-d79e-457b-abd8-9fe67f18fb69-secret-metrics-server-tls\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.487643 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.487600 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtx9t\" (UniqueName: \"kubernetes.io/projected/202b3f86-d79e-457b-abd8-9fe67f18fb69-kube-api-access-dtx9t\") pod \"metrics-server-7479cf6c97-59v8t\" (UID: \"202b3f86-d79e-457b-abd8-9fe67f18fb69\") " pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.574537 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.574499 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:10:57.798358 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:10:57.798143 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36431c2_8598_4a45_b16c_11ac06c52c42.slice/crio-0d3fcbe6b58d7255151225495963701ce1b3d81b68379ca3696b5ba6bb41ecae WatchSource:0}: Error finding container 0d3fcbe6b58d7255151225495963701ce1b3d81b68379ca3696b5ba6bb41ecae: Status 404 returned error can't find the container with id 0d3fcbe6b58d7255151225495963701ce1b3d81b68379ca3696b5ba6bb41ecae Jul 16 21:10:57.881242 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.879750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e4bb1fec-1520-4f19-ba3d-c95eb209e8dd-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-gddmg\" (UID: \"e4bb1fec-1520-4f19-ba3d-c95eb209e8dd\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg" Jul 16 21:10:57.889051 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:57.889004 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e4bb1fec-1520-4f19-ba3d-c95eb209e8dd-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-gddmg\" (UID: \"e4bb1fec-1520-4f19-ba3d-c95eb209e8dd\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg" Jul 16 21:10:58.042005 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.041974 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f5cff5975-jsgmr"] Jul 16 21:10:58.049540 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.049513 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7479cf6c97-59v8t"] Jul 16 21:10:58.058745 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:10:58.058721 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7eb372d_e1e1_41c4_97c8_14bfd1c988c9.slice/crio-3de22ee2d69b1f7f8bbaf8f7f053e2aeda9ca652eaa1af77694f1bb5bd0c9138 WatchSource:0}: Error finding container 3de22ee2d69b1f7f8bbaf8f7f053e2aeda9ca652eaa1af77694f1bb5bd0c9138: Status 404 returned error can't find the container with id 3de22ee2d69b1f7f8bbaf8f7f053e2aeda9ca652eaa1af77694f1bb5bd0c9138 Jul 16 21:10:58.116772 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.116581 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg" Jul 16 21:10:58.252303 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.252276 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg"] Jul 16 21:10:58.254580 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:10:58.254551 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4bb1fec_1520_4f19_ba3d_c95eb209e8dd.slice/crio-95da52b7e471ddd85a19b73137fbec174bfb22e9604323163aaca7a09da11459 WatchSource:0}: Error finding container 95da52b7e471ddd85a19b73137fbec174bfb22e9604323163aaca7a09da11459: Status 404 returned error can't find the container with id 95da52b7e471ddd85a19b73137fbec174bfb22e9604323163aaca7a09da11459 Jul 16 21:10:58.286654 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.286625 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-mqs5r"] Jul 16 21:10:58.289909 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:10:58.289886 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764206ea_3f09_45a2_aae6_47dd7908d230.slice/crio-95d67bbcb6a96db896a09ed429d2e303a35bdebbda3582916de64b034b5ef329 WatchSource:0}: Error finding container 95d67bbcb6a96db896a09ed429d2e303a35bdebbda3582916de64b034b5ef329: Status 404 returned error can't find the container with id 95d67bbcb6a96db896a09ed429d2e303a35bdebbda3582916de64b034b5ef329 Jul 16 21:10:58.296397 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.296374 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:10:58.298746 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:10:58.298721 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb17380_24e0_4749_93ed_48ce0e0dafb2.slice/crio-baa3ac2270cb08e86630f66a7606f032cd8ac86b8865c24b42265b000cbb7e6b WatchSource:0}: Error finding container baa3ac2270cb08e86630f66a7606f032cd8ac86b8865c24b42265b000cbb7e6b: Status 404 returned error can't find the container with id baa3ac2270cb08e86630f66a7606f032cd8ac86b8865c24b42265b000cbb7e6b Jul 16 21:10:58.759535 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.759504 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dbknf" event={"ID":"d278255e-2c35-4471-bbd9-095d9c476896","Type":"ContainerStarted","Data":"43d2369d5fee443481db46b2ffe55f2d409403ad4322192918e0e11249146443"} Jul 16 21:10:58.759662 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.759577 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:10:58.760913 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.760882 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-klcvk" event={"ID":"a36431c2-8598-4a45-b16c-11ac06c52c42","Type":"ContainerStarted","Data":"0d3fcbe6b58d7255151225495963701ce1b3d81b68379ca3696b5ba6bb41ecae"} Jul 16 21:10:58.762918 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.762874 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" event={"ID":"cf548327-8c0f-4961-8c7b-d547ff8a9426","Type":"ContainerStarted","Data":"9ed1ac29fef33e415118412f46176ca600f0cca1804f833f9a50b2df21be4e0c"} Jul 16 21:10:58.763037 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.762962 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" event={"ID":"cf548327-8c0f-4961-8c7b-d547ff8a9426","Type":"ContainerStarted","Data":"acaa3a7f901d53ef2662cac2d8c436229b61f08f5ff9d9d361cbc9e49fd05b2f"} Jul 16 21:10:58.764234 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.764208 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" event={"ID":"202b3f86-d79e-457b-abd8-9fe67f18fb69","Type":"ContainerStarted","Data":"00c1312fff6d865ff2f75c2ecea00e824253616aa46ce37c4371843200159bfb"} Jul 16 21:10:58.766485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.766461 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pjlc6" event={"ID":"f4e81032-aa8b-4011-a300-24472e2b6ab1","Type":"ContainerStarted","Data":"97eb89205d3494c954fe8498e2368db92fb52cdd7d74431d4524bdd870acd9db"} Jul 16 21:10:58.768013 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.767957 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2xbp6" event={"ID":"3bd19c30-8019-4896-aa14-62214c3e042c","Type":"ContainerStarted","Data":"d7e12c336d76b3bdd3dc12d715e95bacdc98e05d80b0292b9e3644dca66f6c8a"} Jul 16 21:10:58.769315 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.769289 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f5cff5975-jsgmr" event={"ID":"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9","Type":"ContainerStarted","Data":"3de22ee2d69b1f7f8bbaf8f7f053e2aeda9ca652eaa1af77694f1bb5bd0c9138"} Jul 16 21:10:58.770674 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.770651 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fb17380-24e0-4749-93ed-48ce0e0dafb2","Type":"ContainerStarted","Data":"baa3ac2270cb08e86630f66a7606f032cd8ac86b8865c24b42265b000cbb7e6b"} Jul 16 21:10:58.772624 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.772598 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-twm75" event={"ID":"9a4e202f-99dc-4068-9496-acb28f0552b5","Type":"ContainerStarted","Data":"62c730204b1199fe6762a116cde8f02ad40edf05583a032f96499c46e47be9d3"} Jul 16 21:10:58.773972 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.773927 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg" event={"ID":"e4bb1fec-1520-4f19-ba3d-c95eb209e8dd","Type":"ContainerStarted","Data":"95da52b7e471ddd85a19b73137fbec174bfb22e9604323163aaca7a09da11459"} Jul 16 21:10:58.775345 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.775326 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" event={"ID":"764206ea-3f09-45a2-aae6-47dd7908d230","Type":"ContainerStarted","Data":"95d67bbcb6a96db896a09ed429d2e303a35bdebbda3582916de64b034b5ef329"} Jul 16 21:10:58.778860 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.778808 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dbknf" podStartSLOduration=68.417437465 podStartE2EDuration="1m14.778787632s" totalImagesPullingTime="6.361350167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:58.777224495 +0000 UTC m=+74.952332710" watchObservedRunningTime="2026-07-16 21:10:58.778787632 +0000 UTC m=+74.953895824" Jul 16 21:10:58.803994 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.803954 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pjlc6" podStartSLOduration=5.172195345 podStartE2EDuration="19.80394103s" totalImagesPullingTime="14.631745685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:10:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:58.803501646 +0000 UTC m=+74.978609859" watchObservedRunningTime="2026-07-16 21:10:58.80394103 +0000 UTC m=+74.979049221" Jul 16 21:10:58.830178 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.830135 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2xbp6" podStartSLOduration=68.448913837 podStartE2EDuration="1m14.830117045s" totalImagesPullingTime="6.381203208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:58.828643454 +0000 UTC m=+75.003751675" watchObservedRunningTime="2026-07-16 21:10:58.830117045 +0000 UTC m=+75.005225239" Jul 16 21:10:58.865488 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:58.865432 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-twm75" podStartSLOduration=72.945587235 podStartE2EDuration="1m14.865411183s" totalImagesPullingTime="1.919823948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:58.864104946 +0000 UTC m=+75.039213139" watchObservedRunningTime="2026-07-16 21:10:58.865411183 +0000 UTC m=+75.040519377" Jul 16 21:10:59.781628 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:59.781585 2577 generic.go:356] "Generic (PLEG): container finished" podID="a36431c2-8598-4a45-b16c-11ac06c52c42" containerID="485236257fa8fd38f73c2da3fdccd8533029880c747482b7b9a04ffc0f897266" exitCode=0 Jul 16 21:10:59.782063 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:10:59.781764 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-klcvk" event={"ID":"a36431c2-8598-4a45-b16c-11ac06c52c42","Type":"ContainerDied","Data":"485236257fa8fd38f73c2da3fdccd8533029880c747482b7b9a04ffc0f897266"} Jul 16 21:11:00.471146 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.471117 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65dbbdbc54-cn8l8"] Jul 16 21:11:00.473996 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.473975 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.487950 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.487926 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65dbbdbc54-cn8l8"] Jul 16 21:11:00.512388 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.512356 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-oauth-serving-cert\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.512525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.512398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2hhs\" (UniqueName: \"kubernetes.io/projected/8bcf6073-6a76-49c5-9133-0edc8f99578e-kube-api-access-m2hhs\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.512525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.512454 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-config\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.512525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.512499 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-trusted-ca-bundle\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.512653 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.512606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-oauth-config\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.512653 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.512630 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-service-ca\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.512756 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.512690 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-serving-cert\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.613809 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.613777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-oauth-config\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.613986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.613823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-service-ca\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.613986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.613866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-serving-cert\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.613986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.613890 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-oauth-serving-cert\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.613986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.613905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2hhs\" (UniqueName: \"kubernetes.io/projected/8bcf6073-6a76-49c5-9133-0edc8f99578e-kube-api-access-m2hhs\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.613986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.613936 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-config\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.613986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.613959 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-trusted-ca-bundle\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.614776 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.614737 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-oauth-serving-cert\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.614906 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.614771 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-config\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.614906 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.614790 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-service-ca\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.614906 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.614883 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-trusted-ca-bundle\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.616552 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.616532 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-oauth-config\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.616822 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.616797 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-serving-cert\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.626623 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.626601 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2hhs\" (UniqueName: \"kubernetes.io/projected/8bcf6073-6a76-49c5-9133-0edc8f99578e-kube-api-access-m2hhs\") pod \"console-65dbbdbc54-cn8l8\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:00.785726 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:00.785656 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:01.676265 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:01.676233 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cd4df479d-m2fwl" Jul 16 21:11:02.220343 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.220312 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65dbbdbc54-cn8l8"] Jul 16 21:11:02.230002 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:11:02.229958 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcf6073_6a76_49c5_9133_0edc8f99578e.slice/crio-189c85e08fae3ead2bd8cac5638a9ff9c3b7f6d33bfd714b4f2c630ad3ce8e8f WatchSource:0}: Error finding container 189c85e08fae3ead2bd8cac5638a9ff9c3b7f6d33bfd714b4f2c630ad3ce8e8f: Status 404 returned error can't find the container with id 189c85e08fae3ead2bd8cac5638a9ff9c3b7f6d33bfd714b4f2c630ad3ce8e8f Jul 16 21:11:02.796416 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.796373 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-klcvk" event={"ID":"a36431c2-8598-4a45-b16c-11ac06c52c42","Type":"ContainerStarted","Data":"1d96f73781f0bf60f5b81880e44783c7e22bb277d4a3aca8cfb2bf3d94609f81"} Jul 16 21:11:02.796583 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.796441 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-klcvk" event={"ID":"a36431c2-8598-4a45-b16c-11ac06c52c42","Type":"ContainerStarted","Data":"6df2b31ae130d94f863f94583e8c985e412f15e734ea558e802ed0b7826e7f34"} Jul 16 21:11:02.798626 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.798592 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" event={"ID":"cf548327-8c0f-4961-8c7b-d547ff8a9426","Type":"ContainerStarted","Data":"2003ebe1f7752b49bf1348a38928db5ad1d019eb010d75343f686efb9b5d1ca2"} Jul 16 21:11:02.800143 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.800118 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" event={"ID":"202b3f86-d79e-457b-abd8-9fe67f18fb69","Type":"ContainerStarted","Data":"2364c046f4b4782183c5d64ee7eaccdd75419fee933de179218aa37b43d2486a"} Jul 16 21:11:02.801655 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.801631 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f5cff5975-jsgmr" event={"ID":"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9","Type":"ContainerStarted","Data":"af055b02d8107b508609cee48193f50e6ec846e2cb76c041fab70dc3627b5952"} Jul 16 21:11:02.803134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.803110 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerID="c5e5d95951480cb794a4488d33573f847480f51bf8e4f1d969c6e7fa917e5640" exitCode=0 Jul 16 21:11:02.803240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.803166 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fb17380-24e0-4749-93ed-48ce0e0dafb2","Type":"ContainerDied","Data":"c5e5d95951480cb794a4488d33573f847480f51bf8e4f1d969c6e7fa917e5640"} Jul 16 21:11:02.804744 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.804722 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg" event={"ID":"e4bb1fec-1520-4f19-ba3d-c95eb209e8dd","Type":"ContainerStarted","Data":"6bed12082bdf26b15a76633f990dbaea2c792576ef59eb31021aa4a7146a3020"} Jul 16 21:11:02.804985 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.804933 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg" Jul 16 21:11:02.807296 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.807271 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" event={"ID":"764206ea-3f09-45a2-aae6-47dd7908d230","Type":"ContainerStarted","Data":"37f39ed9e4ba541282641d64409f530fbea4e428c59871a4e9f052b989b7617e"} Jul 16 21:11:02.807419 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.807328 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" event={"ID":"764206ea-3f09-45a2-aae6-47dd7908d230","Type":"ContainerStarted","Data":"39617fa657bafd0988c6e4f4e9a772b3545e8c6aa09ff7f898d4dd5efd0507d9"} Jul 16 21:11:02.807419 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.807355 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" event={"ID":"764206ea-3f09-45a2-aae6-47dd7908d230","Type":"ContainerStarted","Data":"2fde5dd929747c89ab34cec3637452c9e9aa86ceed8bb939b32bba3dde10a7de"} Jul 16 21:11:02.808927 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.808905 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dbbdbc54-cn8l8" event={"ID":"8bcf6073-6a76-49c5-9133-0edc8f99578e","Type":"ContainerStarted","Data":"2be77c6f111ad8a68df78a172656bbf67ac687931cafc43c2a5ffcb8f0adcce3"} Jul 16 21:11:02.809031 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.808946 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dbbdbc54-cn8l8" event={"ID":"8bcf6073-6a76-49c5-9133-0edc8f99578e","Type":"ContainerStarted","Data":"189c85e08fae3ead2bd8cac5638a9ff9c3b7f6d33bfd714b4f2c630ad3ce8e8f"} Jul 16 21:11:02.813217 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.813178 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg" Jul 16 21:11:02.819633 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.819584 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-klcvk" podStartSLOduration=9.788071432 podStartE2EDuration="10.819549595s" totalImagesPullingTime="956.32319ms" totalInitContainerRuntime="75.154973ms" isStatefulPod=true podCreationTimestamp="2026-07-16 21:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:11:02.81762356 +0000 UTC m=+78.992731752" watchObservedRunningTime="2026-07-16 21:11:02.819549595 +0000 UTC m=+78.994657781" Jul 16 21:11:02.840505 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.840454 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c5b5n" podStartSLOduration=6.839362487 podStartE2EDuration="10.840439347s" totalImagesPullingTime="4.00107686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:11:02.839668111 +0000 UTC m=+79.014776304" watchObservedRunningTime="2026-07-16 21:11:02.840439347 +0000 UTC m=+79.015547538" Jul 16 21:11:02.860421 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.859814 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" podStartSLOduration=1.838351327 podStartE2EDuration="5.859798331s" totalImagesPullingTime="4.021447004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:11:02.858488323 +0000 UTC m=+79.033596549" watchObservedRunningTime="2026-07-16 21:11:02.859798331 +0000 UTC m=+79.034906525" Jul 16 21:11:02.877801 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.877763 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65dbbdbc54-cn8l8" podStartSLOduration=2.877750699 podStartE2EDuration="2.877750699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:11:02.877152979 +0000 UTC m=+79.052261209" watchObservedRunningTime="2026-07-16 21:11:02.877750699 +0000 UTC m=+79.052858890" Jul 16 21:11:02.929324 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.929283 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f5cff5975-jsgmr" podStartSLOduration=3.85370634 podStartE2EDuration="7.92927064s" totalImagesPullingTime="4.0755643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:11:02.928066936 +0000 UTC m=+79.103175153" watchObservedRunningTime="2026-07-16 21:11:02.92927064 +0000 UTC m=+79.104378831" Jul 16 21:11:02.947198 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.947137 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-mqs5r" podStartSLOduration=7.15736803 podStartE2EDuration="10.947120806s" totalImagesPullingTime="3.789752776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:11:02.946935937 +0000 UTC m=+79.122044183" watchObservedRunningTime="2026-07-16 21:11:02.947120806 +0000 UTC m=+79.122228999" Jul 16 21:11:02.966369 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:02.966326 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-gddmg" podStartSLOduration=2.138925751 podStartE2EDuration="5.966313417s" totalImagesPullingTime="3.827387666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:11:02.965482267 +0000 UTC m=+79.140590458" watchObservedRunningTime="2026-07-16 21:11:02.966313417 +0000 UTC m=+79.141421623" Jul 16 21:11:04.819775 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:04.819747 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fb17380-24e0-4749-93ed-48ce0e0dafb2","Type":"ContainerStarted","Data":"53cca51d3f8704f3f885bab1994fa2534db8b19d6fda3bcae8a7c58feb91fedc"} Jul 16 21:11:04.820063 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:04.819812 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fb17380-24e0-4749-93ed-48ce0e0dafb2","Type":"ContainerStarted","Data":"8aa3f62e66139bfa5e37781256a3ba999f359f50badfa960c05ff0aa4255783f"} Jul 16 21:11:04.820063 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:04.819838 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fb17380-24e0-4749-93ed-48ce0e0dafb2","Type":"ContainerStarted","Data":"f5b7075dde337cbf34267d4e64fa6e9fb97ba9f9a037b99df8e984892ff3119e"} Jul 16 21:11:05.827207 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:05.827155 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fb17380-24e0-4749-93ed-48ce0e0dafb2","Type":"ContainerStarted","Data":"522cef7289ceef57dcb50b421ab440906704d9f3d803707f7fa0c03729124f36"} Jul 16 21:11:05.827504 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:05.827258 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fb17380-24e0-4749-93ed-48ce0e0dafb2","Type":"ContainerStarted","Data":"c1f9600584d54f143ec3e18f00e03aedc318ea3c2e5082d2d60c1607c565c0cb"} Jul 16 21:11:05.898495 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:05.898458 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:11:05.898618 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:05.898525 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:11:05.903110 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:05.903092 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:11:06.835741 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:06.835703 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fb17380-24e0-4749-93ed-48ce0e0dafb2","Type":"ContainerStarted","Data":"13536b44965aad0576ace83cf7ebbad98126613e616560e7c084b11afb234c00"} Jul 16 21:11:06.839719 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:06.839695 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:11:06.877427 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:06.877385 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=7.355356164 podStartE2EDuration="13.877371202s" totalImagesPullingTime="6.486362273s" totalInitContainerRuntime="35.652765ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:10:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:11:06.872086113 +0000 UTC m=+83.047194305" watchObservedRunningTime="2026-07-16 21:11:06.877371202 +0000 UTC m=+83.052479423" Jul 16 21:11:10.786214 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:10.786166 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:10.786671 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:10.786246 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:10.790874 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:10.790853 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:10.854328 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:10.854300 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:11:10.943595 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:10.943561 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f5cff5975-jsgmr"] Jul 16 21:11:17.575611 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:17.575579 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:11:17.575959 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:17.575636 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:11:29.784856 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:29.784831 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dbknf" Jul 16 21:11:35.964224 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:35.964147 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f5cff5975-jsgmr" podUID="a7eb372d-e1e1-41c4-97c8-14bfd1c988c9" containerName="console" containerID="cri-o://af055b02d8107b508609cee48193f50e6ec846e2cb76c041fab70dc3627b5952" gracePeriod=15 Jul 16 21:11:36.259492 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.259468 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:11:36.378366 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.378342 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-trusted-ca-bundle\") pod \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " Jul 16 21:11:36.378498 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.378372 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-service-ca\") pod \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " Jul 16 21:11:36.378498 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.378419 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-config\") pod \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " Jul 16 21:11:36.378498 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.378440 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-serving-cert\") pod \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " Jul 16 21:11:36.378647 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.378625 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-oauth-config\") pod \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " Jul 16 21:11:36.378698 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.378675 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-oauth-serving-cert\") pod \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " Jul 16 21:11:36.378751 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.378710 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m67wg\" (UniqueName: \"kubernetes.io/projected/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-kube-api-access-m67wg\") pod \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\" (UID: \"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9\") " Jul 16 21:11:36.378804 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.378779 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-config" (OuterVolumeSpecName: "console-config") pod "a7eb372d-e1e1-41c4-97c8-14bfd1c988c9" (UID: "a7eb372d-e1e1-41c4-97c8-14bfd1c988c9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:11:36.378804 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.378787 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a7eb372d-e1e1-41c4-97c8-14bfd1c988c9" (UID: "a7eb372d-e1e1-41c4-97c8-14bfd1c988c9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:11:36.378901 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.378854 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-service-ca" (OuterVolumeSpecName: "service-ca") pod "a7eb372d-e1e1-41c4-97c8-14bfd1c988c9" (UID: "a7eb372d-e1e1-41c4-97c8-14bfd1c988c9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:11:36.379079 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.379052 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a7eb372d-e1e1-41c4-97c8-14bfd1c988c9" (UID: "a7eb372d-e1e1-41c4-97c8-14bfd1c988c9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:11:36.379169 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.379118 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-trusted-ca-bundle\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:11:36.379169 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.379137 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:11:36.379169 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.379150 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-config\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:11:36.379169 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.379158 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-oauth-serving-cert\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:11:36.380746 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.380718 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a7eb372d-e1e1-41c4-97c8-14bfd1c988c9" (UID: "a7eb372d-e1e1-41c4-97c8-14bfd1c988c9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:11:36.380833 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.380756 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-kube-api-access-m67wg" (OuterVolumeSpecName: "kube-api-access-m67wg") pod "a7eb372d-e1e1-41c4-97c8-14bfd1c988c9" (UID: "a7eb372d-e1e1-41c4-97c8-14bfd1c988c9"). InnerVolumeSpecName "kube-api-access-m67wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:11:36.380833 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.380775 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a7eb372d-e1e1-41c4-97c8-14bfd1c988c9" (UID: "a7eb372d-e1e1-41c4-97c8-14bfd1c988c9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:11:36.479825 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.479773 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-serving-cert\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:11:36.479825 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.479796 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-console-oauth-config\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:11:36.479825 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.479806 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m67wg\" (UniqueName: \"kubernetes.io/projected/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9-kube-api-access-m67wg\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:11:36.950030 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.949998 2577 generic.go:356] "Generic (PLEG): container finished" podID="a7eb372d-e1e1-41c4-97c8-14bfd1c988c9" containerID="af055b02d8107b508609cee48193f50e6ec846e2cb76c041fab70dc3627b5952" exitCode=0 Jul 16 21:11:36.950289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.950044 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f5cff5975-jsgmr" Jul 16 21:11:36.950289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.950087 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f5cff5975-jsgmr" event={"ID":"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9","Type":"ContainerDied","Data":"af055b02d8107b508609cee48193f50e6ec846e2cb76c041fab70dc3627b5952"} Jul 16 21:11:36.950289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.950140 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f5cff5975-jsgmr" event={"ID":"a7eb372d-e1e1-41c4-97c8-14bfd1c988c9","Type":"ContainerDied","Data":"3de22ee2d69b1f7f8bbaf8f7f053e2aeda9ca652eaa1af77694f1bb5bd0c9138"} Jul 16 21:11:36.950289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.950162 2577 scope.go:117] "RemoveContainer" containerID="af055b02d8107b508609cee48193f50e6ec846e2cb76c041fab70dc3627b5952" Jul 16 21:11:36.958094 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.958077 2577 scope.go:117] "RemoveContainer" containerID="af055b02d8107b508609cee48193f50e6ec846e2cb76c041fab70dc3627b5952" Jul 16 21:11:36.958418 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:11:36.958396 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af055b02d8107b508609cee48193f50e6ec846e2cb76c041fab70dc3627b5952\": container with ID starting with af055b02d8107b508609cee48193f50e6ec846e2cb76c041fab70dc3627b5952 not found: ID does not exist" containerID="af055b02d8107b508609cee48193f50e6ec846e2cb76c041fab70dc3627b5952" Jul 16 21:11:36.958468 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.958430 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af055b02d8107b508609cee48193f50e6ec846e2cb76c041fab70dc3627b5952"} err="failed to get container status \"af055b02d8107b508609cee48193f50e6ec846e2cb76c041fab70dc3627b5952\": rpc error: code = NotFound desc = could not find container \"af055b02d8107b508609cee48193f50e6ec846e2cb76c041fab70dc3627b5952\": container with ID starting with af055b02d8107b508609cee48193f50e6ec846e2cb76c041fab70dc3627b5952 not found: ID does not exist" Jul 16 21:11:36.969707 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.969688 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f5cff5975-jsgmr"] Jul 16 21:11:36.975754 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:36.975733 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f5cff5975-jsgmr"] Jul 16 21:11:37.580113 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:37.580088 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:11:37.583989 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:37.583970 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7479cf6c97-59v8t" Jul 16 21:11:38.449664 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:11:38.449620 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7eb372d-e1e1-41c4-97c8-14bfd1c988c9" path="/var/lib/kubelet/pods/a7eb372d-e1e1-41c4-97c8-14bfd1c988c9/volumes" Jul 16 21:12:12.916805 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:12.916768 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:12:12.917257 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:12.917046 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="alertmanager" containerID="cri-o://f5b7075dde337cbf34267d4e64fa6e9fb97ba9f9a037b99df8e984892ff3119e" gracePeriod=120 Jul 16 21:12:12.917257 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:12.917119 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="kube-rbac-proxy-metric" containerID="cri-o://522cef7289ceef57dcb50b421ab440906704d9f3d803707f7fa0c03729124f36" gracePeriod=120 Jul 16 21:12:12.917257 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:12.917131 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="kube-rbac-proxy" containerID="cri-o://c1f9600584d54f143ec3e18f00e03aedc318ea3c2e5082d2d60c1607c565c0cb" gracePeriod=120 Jul 16 21:12:12.917257 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:12.917164 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="config-reloader" containerID="cri-o://8aa3f62e66139bfa5e37781256a3ba999f359f50badfa960c05ff0aa4255783f" gracePeriod=120 Jul 16 21:12:12.917257 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:12.917121 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="kube-rbac-proxy-web" containerID="cri-o://53cca51d3f8704f3f885bab1994fa2534db8b19d6fda3bcae8a7c58feb91fedc" gracePeriod=120 Jul 16 21:12:12.917510 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:12.917230 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="prom-label-proxy" containerID="cri-o://13536b44965aad0576ace83cf7ebbad98126613e616560e7c084b11afb234c00" gracePeriod=120 Jul 16 21:12:13.081175 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:13.081148 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerID="13536b44965aad0576ace83cf7ebbad98126613e616560e7c084b11afb234c00" exitCode=0 Jul 16 21:12:13.081175 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:13.081171 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerID="c1f9600584d54f143ec3e18f00e03aedc318ea3c2e5082d2d60c1607c565c0cb" exitCode=0 Jul 16 21:12:13.081175 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:13.081177 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerID="8aa3f62e66139bfa5e37781256a3ba999f359f50badfa960c05ff0aa4255783f" exitCode=0 Jul 16 21:12:13.081402 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:13.081197 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerID="f5b7075dde337cbf34267d4e64fa6e9fb97ba9f9a037b99df8e984892ff3119e" exitCode=0 Jul 16 21:12:13.081402 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:13.081215 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fb17380-24e0-4749-93ed-48ce0e0dafb2","Type":"ContainerDied","Data":"13536b44965aad0576ace83cf7ebbad98126613e616560e7c084b11afb234c00"} Jul 16 21:12:13.081402 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:13.081296 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fb17380-24e0-4749-93ed-48ce0e0dafb2","Type":"ContainerDied","Data":"c1f9600584d54f143ec3e18f00e03aedc318ea3c2e5082d2d60c1607c565c0cb"} Jul 16 21:12:13.081402 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:13.081341 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fb17380-24e0-4749-93ed-48ce0e0dafb2","Type":"ContainerDied","Data":"8aa3f62e66139bfa5e37781256a3ba999f359f50badfa960c05ff0aa4255783f"} Jul 16 21:12:13.081402 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:13.081384 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fb17380-24e0-4749-93ed-48ce0e0dafb2","Type":"ContainerDied","Data":"f5b7075dde337cbf34267d4e64fa6e9fb97ba9f9a037b99df8e984892ff3119e"} Jul 16 21:12:14.093250 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.093222 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerID="522cef7289ceef57dcb50b421ab440906704d9f3d803707f7fa0c03729124f36" exitCode=0 Jul 16 21:12:14.093250 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.093248 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerID="53cca51d3f8704f3f885bab1994fa2534db8b19d6fda3bcae8a7c58feb91fedc" exitCode=0 Jul 16 21:12:14.093582 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.093297 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fb17380-24e0-4749-93ed-48ce0e0dafb2","Type":"ContainerDied","Data":"522cef7289ceef57dcb50b421ab440906704d9f3d803707f7fa0c03729124f36"} Jul 16 21:12:14.093582 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.093408 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fb17380-24e0-4749-93ed-48ce0e0dafb2","Type":"ContainerDied","Data":"53cca51d3f8704f3f885bab1994fa2534db8b19d6fda3bcae8a7c58feb91fedc"} Jul 16 21:12:14.158638 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.158617 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:14.248043 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.247970 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " Jul 16 21:12:14.248043 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.248006 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-config-volume\") pod \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " Jul 16 21:12:14.248043 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.248029 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fb17380-24e0-4749-93ed-48ce0e0dafb2-config-out\") pod \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " Jul 16 21:12:14.248302 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.248057 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-web-config\") pod \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " Jul 16 21:12:14.248385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.248357 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-main-tls\") pod \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " Jul 16 21:12:14.248500 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.248414 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fb17380-24e0-4749-93ed-48ce0e0dafb2-alertmanager-trusted-ca-bundle\") pod \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " Jul 16 21:12:14.248500 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.248449 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-cluster-tls-config\") pod \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " Jul 16 21:12:14.248500 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.248489 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8659\" (UniqueName: \"kubernetes.io/projected/9fb17380-24e0-4749-93ed-48ce0e0dafb2-kube-api-access-f8659\") pod \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " Jul 16 21:12:14.248711 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.248528 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9fb17380-24e0-4749-93ed-48ce0e0dafb2-alertmanager-main-db\") pod \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " Jul 16 21:12:14.248711 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.248564 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy\") pod \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " Jul 16 21:12:14.248711 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.248588 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fb17380-24e0-4749-93ed-48ce0e0dafb2-metrics-client-ca\") pod \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " Jul 16 21:12:14.248711 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.248631 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy-web\") pod \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " Jul 16 21:12:14.248711 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.248666 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fb17380-24e0-4749-93ed-48ce0e0dafb2-tls-assets\") pod \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\" (UID: \"9fb17380-24e0-4749-93ed-48ce0e0dafb2\") " Jul 16 21:12:14.248958 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.248768 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb17380-24e0-4749-93ed-48ce0e0dafb2-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "9fb17380-24e0-4749-93ed-48ce0e0dafb2" (UID: "9fb17380-24e0-4749-93ed-48ce0e0dafb2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:12:14.248958 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.248943 2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fb17380-24e0-4749-93ed-48ce0e0dafb2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:14.250215 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.250168 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb17380-24e0-4749-93ed-48ce0e0dafb2-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "9fb17380-24e0-4749-93ed-48ce0e0dafb2" (UID: "9fb17380-24e0-4749-93ed-48ce0e0dafb2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:12:14.250643 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.250569 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb17380-24e0-4749-93ed-48ce0e0dafb2-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "9fb17380-24e0-4749-93ed-48ce0e0dafb2" (UID: "9fb17380-24e0-4749-93ed-48ce0e0dafb2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:12:14.251954 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.251917 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb17380-24e0-4749-93ed-48ce0e0dafb2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9fb17380-24e0-4749-93ed-48ce0e0dafb2" (UID: "9fb17380-24e0-4749-93ed-48ce0e0dafb2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:12:14.252411 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.252381 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "9fb17380-24e0-4749-93ed-48ce0e0dafb2" (UID: "9fb17380-24e0-4749-93ed-48ce0e0dafb2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:12:14.252579 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.252554 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "9fb17380-24e0-4749-93ed-48ce0e0dafb2" (UID: "9fb17380-24e0-4749-93ed-48ce0e0dafb2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:12:14.252776 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.252727 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb17380-24e0-4749-93ed-48ce0e0dafb2-config-out" (OuterVolumeSpecName: "config-out") pod "9fb17380-24e0-4749-93ed-48ce0e0dafb2" (UID: "9fb17380-24e0-4749-93ed-48ce0e0dafb2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:12:14.252885 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.252828 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb17380-24e0-4749-93ed-48ce0e0dafb2-kube-api-access-f8659" (OuterVolumeSpecName: "kube-api-access-f8659") pod "9fb17380-24e0-4749-93ed-48ce0e0dafb2" (UID: "9fb17380-24e0-4749-93ed-48ce0e0dafb2"). InnerVolumeSpecName "kube-api-access-f8659". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:12:14.253089 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.253002 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "9fb17380-24e0-4749-93ed-48ce0e0dafb2" (UID: "9fb17380-24e0-4749-93ed-48ce0e0dafb2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:12:14.253163 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.253095 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-config-volume" (OuterVolumeSpecName: "config-volume") pod "9fb17380-24e0-4749-93ed-48ce0e0dafb2" (UID: "9fb17380-24e0-4749-93ed-48ce0e0dafb2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:12:14.253247 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.253167 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "9fb17380-24e0-4749-93ed-48ce0e0dafb2" (UID: "9fb17380-24e0-4749-93ed-48ce0e0dafb2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:12:14.256145 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.256101 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "9fb17380-24e0-4749-93ed-48ce0e0dafb2" (UID: "9fb17380-24e0-4749-93ed-48ce0e0dafb2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:12:14.261883 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.261860 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-web-config" (OuterVolumeSpecName: "web-config") pod "9fb17380-24e0-4749-93ed-48ce0e0dafb2" (UID: "9fb17380-24e0-4749-93ed-48ce0e0dafb2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:12:14.349317 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.349281 2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fb17380-24e0-4749-93ed-48ce0e0dafb2-tls-assets\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:14.349317 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.349315 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:14.349317 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.349327 2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-config-volume\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:14.349485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.349337 2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fb17380-24e0-4749-93ed-48ce0e0dafb2-config-out\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:14.349485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.349344 2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-web-config\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:14.349485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.349353 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-main-tls\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:14.349485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.349362 2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-cluster-tls-config\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:14.349485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.349370 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8659\" (UniqueName: \"kubernetes.io/projected/9fb17380-24e0-4749-93ed-48ce0e0dafb2-kube-api-access-f8659\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:14.349485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.349378 2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9fb17380-24e0-4749-93ed-48ce0e0dafb2-alertmanager-main-db\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:14.349485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.349387 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:14.349485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.349395 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fb17380-24e0-4749-93ed-48ce0e0dafb2-metrics-client-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:14.349485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:14.349403 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fb17380-24e0-4749-93ed-48ce0e0dafb2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:15.099980 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.099945 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fb17380-24e0-4749-93ed-48ce0e0dafb2","Type":"ContainerDied","Data":"baa3ac2270cb08e86630f66a7606f032cd8ac86b8865c24b42265b000cbb7e6b"} Jul 16 21:12:15.099980 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.099980 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.100356 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.100029 2577 scope.go:117] "RemoveContainer" containerID="13536b44965aad0576ace83cf7ebbad98126613e616560e7c084b11afb234c00" Jul 16 21:12:15.107414 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.107398 2577 scope.go:117] "RemoveContainer" containerID="522cef7289ceef57dcb50b421ab440906704d9f3d803707f7fa0c03729124f36" Jul 16 21:12:15.113693 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.113677 2577 scope.go:117] "RemoveContainer" containerID="c1f9600584d54f143ec3e18f00e03aedc318ea3c2e5082d2d60c1607c565c0cb" Jul 16 21:12:15.119904 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.119887 2577 scope.go:117] "RemoveContainer" containerID="53cca51d3f8704f3f885bab1994fa2534db8b19d6fda3bcae8a7c58feb91fedc" Jul 16 21:12:15.126427 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.126319 2577 scope.go:117] "RemoveContainer" containerID="8aa3f62e66139bfa5e37781256a3ba999f359f50badfa960c05ff0aa4255783f" Jul 16 21:12:15.127622 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.127605 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:12:15.132891 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.132866 2577 scope.go:117] "RemoveContainer" containerID="f5b7075dde337cbf34267d4e64fa6e9fb97ba9f9a037b99df8e984892ff3119e" Jul 16 21:12:15.136086 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.136071 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:12:15.139711 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.139694 2577 scope.go:117] "RemoveContainer" containerID="c5e5d95951480cb794a4488d33573f847480f51bf8e4f1d969c6e7fa917e5640" Jul 16 21:12:15.167104 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167080 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:12:15.167341 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167329 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="kube-rbac-proxy-metric" Jul 16 21:12:15.167388 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167343 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="kube-rbac-proxy-metric" Jul 16 21:12:15.167388 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167358 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="kube-rbac-proxy-web" Jul 16 21:12:15.167388 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167363 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="kube-rbac-proxy-web" Jul 16 21:12:15.167388 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167371 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="config-reloader" Jul 16 21:12:15.167388 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167376 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="config-reloader" Jul 16 21:12:15.167388 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167386 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="init-config-reloader" Jul 16 21:12:15.167572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167392 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="init-config-reloader" Jul 16 21:12:15.167572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167398 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="kube-rbac-proxy" Jul 16 21:12:15.167572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167403 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="kube-rbac-proxy" Jul 16 21:12:15.167572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167409 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="prom-label-proxy" Jul 16 21:12:15.167572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167414 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="prom-label-proxy" Jul 16 21:12:15.167572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167422 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="alertmanager" Jul 16 21:12:15.167572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167427 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="alertmanager" Jul 16 21:12:15.167572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167433 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7eb372d-e1e1-41c4-97c8-14bfd1c988c9" containerName="console" Jul 16 21:12:15.167572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167438 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7eb372d-e1e1-41c4-97c8-14bfd1c988c9" containerName="console" Jul 16 21:12:15.167572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167478 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="config-reloader" Jul 16 21:12:15.167572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167484 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="kube-rbac-proxy-web" Jul 16 21:12:15.167572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167490 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="prom-label-proxy" Jul 16 21:12:15.167572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167497 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7eb372d-e1e1-41c4-97c8-14bfd1c988c9" containerName="console" Jul 16 21:12:15.167572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167502 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="alertmanager" Jul 16 21:12:15.167572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167507 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="kube-rbac-proxy" Jul 16 21:12:15.167572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.167512 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" containerName="kube-rbac-proxy-metric" Jul 16 21:12:15.171061 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.171045 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.173843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.173819 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 21:12:15.173843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.173840 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 21:12:15.174100 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.174085 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 21:12:15.174365 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.174350 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 21:12:15.174446 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.174368 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 21:12:15.174987 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.174972 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 21:12:15.175055 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.174995 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 21:12:15.175490 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.175476 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8d64h\"" Jul 16 21:12:15.175784 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.175753 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 21:12:15.183399 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.183384 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 21:12:15.192038 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.192019 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:12:15.252894 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.252839 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-web-config\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.252894 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.252867 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.252894 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.252885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.253104 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.252909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.253104 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.252972 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.253104 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.253029 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.253104 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.253057 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.253104 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.253093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-config-out\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.253391 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.253128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.253391 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.253217 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-config-volume\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.253391 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.253249 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg4f2\" (UniqueName: \"kubernetes.io/projected/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-kube-api-access-hg4f2\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.253391 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.253269 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.253391 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.253288 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.353933 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.353906 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.354134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.353950 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-config-volume\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.354134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.353969 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg4f2\" (UniqueName: \"kubernetes.io/projected/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-kube-api-access-hg4f2\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.354134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.353986 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.354134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.354002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.354134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.354022 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-web-config\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.354134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.354037 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.354134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.354052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.354134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.354068 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.354134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.354085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.354134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.354117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.354661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.354140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.354661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.354165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-config-out\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.354767 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.354753 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.354989 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.354970 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.355424 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.355396 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.357280 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.356912 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-config-out\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.357280 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.356914 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.357280 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.356967 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-config-volume\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.357280 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.357063 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.357548 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.357525 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.357710 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.357691 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.357970 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.357954 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-web-config\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.358099 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.358080 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.358839 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.358822 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.363404 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.363379 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg4f2\" (UniqueName: \"kubernetes.io/projected/fecc5f4c-26b0-420e-9c65-1bcba45fb17f-kube-api-access-hg4f2\") pod \"alertmanager-main-0\" (UID: \"fecc5f4c-26b0-420e-9c65-1bcba45fb17f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.479645 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.479608 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:12:15.616724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:15.616701 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:12:15.619090 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:12:15.619062 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecc5f4c_26b0_420e_9c65_1bcba45fb17f.slice/crio-3c1d32b87e7e7bf184b0c624872f82d2264de76a392f3a8013a6f10e7f1196b7 WatchSource:0}: Error finding container 3c1d32b87e7e7bf184b0c624872f82d2264de76a392f3a8013a6f10e7f1196b7: Status 404 returned error can't find the container with id 3c1d32b87e7e7bf184b0c624872f82d2264de76a392f3a8013a6f10e7f1196b7 Jul 16 21:12:16.108152 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:16.108122 2577 generic.go:356] "Generic (PLEG): container finished" podID="fecc5f4c-26b0-420e-9c65-1bcba45fb17f" containerID="a05476e882d7e4036ce76a96daa2410f6f4668852fcd7ac22fbc080545741e0c" exitCode=0 Jul 16 21:12:16.108571 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:16.108210 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fecc5f4c-26b0-420e-9c65-1bcba45fb17f","Type":"ContainerDied","Data":"a05476e882d7e4036ce76a96daa2410f6f4668852fcd7ac22fbc080545741e0c"} Jul 16 21:12:16.108571 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:16.108273 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fecc5f4c-26b0-420e-9c65-1bcba45fb17f","Type":"ContainerStarted","Data":"3c1d32b87e7e7bf184b0c624872f82d2264de76a392f3a8013a6f10e7f1196b7"} Jul 16 21:12:16.449544 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:16.449519 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fb17380-24e0-4749-93ed-48ce0e0dafb2" path="/var/lib/kubelet/pods/9fb17380-24e0-4749-93ed-48ce0e0dafb2/volumes" Jul 16 21:12:16.995596 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:16.995563 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7ccdc67968-mbhw7"] Jul 16 21:12:16.998519 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:16.998492 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.002116 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.002095 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 16 21:12:17.002240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.002149 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 16 21:12:17.002379 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.002366 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 16 21:12:17.003043 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.003029 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 16 21:12:17.003295 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.003282 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 16 21:12:17.003627 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.003612 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lwxbc\"" Jul 16 21:12:17.007094 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.007079 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 16 21:12:17.015554 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.015534 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7ccdc67968-mbhw7"] Jul 16 21:12:17.065235 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.065209 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-secret-telemeter-client\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.065325 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.065249 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggh8c\" (UniqueName: \"kubernetes.io/projected/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-kube-api-access-ggh8c\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.065325 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.065317 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.065402 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.065338 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-serving-certs-ca-bundle\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.065402 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.065364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-telemeter-client-tls\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.065475 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.065413 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-metrics-client-ca\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.065475 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.065436 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.065475 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.065462 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-federate-client-tls\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.116744 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.116718 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fecc5f4c-26b0-420e-9c65-1bcba45fb17f","Type":"ContainerStarted","Data":"c94371dd9340d233bffa4e6369d8f82bc5ba86c4006512b25cd1b4eed09ecfed"} Jul 16 21:12:17.116994 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.116784 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fecc5f4c-26b0-420e-9c65-1bcba45fb17f","Type":"ContainerStarted","Data":"fd156d269df2e4ed0883f77e8e11e693632189d43d97831360ae6ef46a8aa6ca"} Jul 16 21:12:17.116994 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.116825 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fecc5f4c-26b0-420e-9c65-1bcba45fb17f","Type":"ContainerStarted","Data":"8c201f13020ec5d716c046baa33c661b73566a4e71b7a988c96f0a85ca6f16a6"} Jul 16 21:12:17.116994 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.116863 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fecc5f4c-26b0-420e-9c65-1bcba45fb17f","Type":"ContainerStarted","Data":"805397e921f1e722938fcdbd78c3c0908f954bd1b324ecdda60e2df7bc43b21f"} Jul 16 21:12:17.116994 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.116905 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fecc5f4c-26b0-420e-9c65-1bcba45fb17f","Type":"ContainerStarted","Data":"f54177b5c5989240bec5587b1d35d5e303fa6ac43630a34f7b2e46b8621e9947"} Jul 16 21:12:17.116994 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.116942 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fecc5f4c-26b0-420e-9c65-1bcba45fb17f","Type":"ContainerStarted","Data":"14c113ec0376a815d4d70f73bef9c9ddaa91ead001db9f137fb2c7fc9a7786ac"} Jul 16 21:12:17.148320 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.148267 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.134079768 podStartE2EDuration="2.148253111s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.173343ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:12:17.146976985 +0000 UTC m=+153.322085199" watchObservedRunningTime="2026-07-16 21:12:17.148253111 +0000 UTC m=+153.323361312" Jul 16 21:12:17.166649 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.166624 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.166749 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.166661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-federate-client-tls\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.166749 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.166708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-secret-telemeter-client\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.166870 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.166750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggh8c\" (UniqueName: \"kubernetes.io/projected/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-kube-api-access-ggh8c\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.166870 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.166821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.167021 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.166903 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-serving-certs-ca-bundle\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.167415 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.167394 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-telemeter-client-tls\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.167535 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.167516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-metrics-client-ca\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.168491 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.168458 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.169845 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.169820 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-metrics-client-ca\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.171277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.171254 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-serving-certs-ca-bundle\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.176115 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.176087 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-secret-telemeter-client\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.176682 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.176660 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-telemeter-client-tls\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.177231 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.177210 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.177378 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.177356 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-federate-client-tls\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.177596 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.177563 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggh8c\" (UniqueName: \"kubernetes.io/projected/9c6934bf-761d-4d75-86e3-8f74bdb8ffb9-kube-api-access-ggh8c\") pod \"telemeter-client-7ccdc67968-mbhw7\" (UID: \"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9\") " pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.308611 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.308542 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" Jul 16 21:12:17.441546 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:17.441523 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7ccdc67968-mbhw7"] Jul 16 21:12:17.443761 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:12:17.443732 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6934bf_761d_4d75_86e3_8f74bdb8ffb9.slice/crio-f340917e18707d84d40497478ffcdeed3df9b864b15605eeec109222c6fa63be WatchSource:0}: Error finding container f340917e18707d84d40497478ffcdeed3df9b864b15605eeec109222c6fa63be: Status 404 returned error can't find the container with id f340917e18707d84d40497478ffcdeed3df9b864b15605eeec109222c6fa63be Jul 16 21:12:18.121248 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:18.121217 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" event={"ID":"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9","Type":"ContainerStarted","Data":"f340917e18707d84d40497478ffcdeed3df9b864b15605eeec109222c6fa63be"} Jul 16 21:12:21.137416 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:21.137382 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" event={"ID":"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9","Type":"ContainerStarted","Data":"8b9016de03bbdcf6f2e6096c1f5c15cb22d355de02a44a0dc256a02cd99f43f1"} Jul 16 21:12:21.137840 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:21.137445 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" event={"ID":"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9","Type":"ContainerStarted","Data":"ba7546f504595c484ee60ffaca16b9feb4bfbadfe8d8e85b71eac911894fae44"} Jul 16 21:12:21.137840 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:21.137480 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" event={"ID":"9c6934bf-761d-4d75-86e3-8f74bdb8ffb9","Type":"ContainerStarted","Data":"a1e04372880607d3e0c7f21f3ea11101c99e91681b5b19770d5e07ac36951e3a"} Jul 16 21:12:21.168554 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:21.167005 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7ccdc67968-mbhw7" podStartSLOduration=2.453899065 podStartE2EDuration="5.166985519s" totalImagesPullingTime="2.713086454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:12:21.164959188 +0000 UTC m=+157.340067402" watchObservedRunningTime="2026-07-16 21:12:21.166985519 +0000 UTC m=+157.342093716" Jul 16 21:12:22.121435 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.121408 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-574f89c6f4-ck4mn"] Jul 16 21:12:22.123811 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.123795 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.136828 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.136802 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-574f89c6f4-ck4mn"] Jul 16 21:12:22.207378 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.207354 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f34028c-6657-405d-966a-1f9292185a33-console-serving-cert\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.207741 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.207414 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-oauth-serving-cert\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.207741 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.207455 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-console-config\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.207741 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.207492 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8jb9\" (UniqueName: \"kubernetes.io/projected/9f34028c-6657-405d-966a-1f9292185a33-kube-api-access-m8jb9\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.207741 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.207676 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-service-ca\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.207937 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.207793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f34028c-6657-405d-966a-1f9292185a33-console-oauth-config\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.207937 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.207824 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-trusted-ca-bundle\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.308146 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.308119 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-service-ca\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.308289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.308154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f34028c-6657-405d-966a-1f9292185a33-console-oauth-config\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.308289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.308169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-trusted-ca-bundle\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.308289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.308218 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f34028c-6657-405d-966a-1f9292185a33-console-serving-cert\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.308289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.308272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-oauth-serving-cert\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.308515 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.308301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-console-config\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.308515 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.308336 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jb9\" (UniqueName: \"kubernetes.io/projected/9f34028c-6657-405d-966a-1f9292185a33-kube-api-access-m8jb9\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.308908 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.308882 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-service-ca\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.309018 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.308991 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-oauth-serving-cert\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.309110 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.309057 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-trusted-ca-bundle\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.309157 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.309111 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-console-config\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.310657 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.310637 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f34028c-6657-405d-966a-1f9292185a33-console-serving-cert\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.310812 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.310794 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f34028c-6657-405d-966a-1f9292185a33-console-oauth-config\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.317695 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.317676 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jb9\" (UniqueName: \"kubernetes.io/projected/9f34028c-6657-405d-966a-1f9292185a33-kube-api-access-m8jb9\") pod \"console-574f89c6f4-ck4mn\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.432087 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.432067 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:22.557417 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:22.557380 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-574f89c6f4-ck4mn"] Jul 16 21:12:22.560466 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:12:22.560437 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f34028c_6657_405d_966a_1f9292185a33.slice/crio-49e820dbb6cb67789a4398d479bf95fef1586f5cf0b3db443423ab10d866504d WatchSource:0}: Error finding container 49e820dbb6cb67789a4398d479bf95fef1586f5cf0b3db443423ab10d866504d: Status 404 returned error can't find the container with id 49e820dbb6cb67789a4398d479bf95fef1586f5cf0b3db443423ab10d866504d Jul 16 21:12:23.147127 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:23.147097 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574f89c6f4-ck4mn" event={"ID":"9f34028c-6657-405d-966a-1f9292185a33","Type":"ContainerStarted","Data":"20e104f3151bd482dd2faa20a935da21b7edf5e6fee909387656f48f92ba8091"} Jul 16 21:12:23.147317 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:23.147147 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574f89c6f4-ck4mn" event={"ID":"9f34028c-6657-405d-966a-1f9292185a33","Type":"ContainerStarted","Data":"49e820dbb6cb67789a4398d479bf95fef1586f5cf0b3db443423ab10d866504d"} Jul 16 21:12:23.169328 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:23.169278 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-574f89c6f4-ck4mn" podStartSLOduration=1.169260905 podStartE2EDuration="1.169260905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:12:23.167926207 +0000 UTC m=+159.343034441" watchObservedRunningTime="2026-07-16 21:12:23.169260905 +0000 UTC m=+159.344369098" Jul 16 21:12:32.433004 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:32.432966 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:32.433392 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:32.433054 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:32.437655 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:32.437633 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:33.188559 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:33.188529 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:12:33.241700 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:33.241662 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65dbbdbc54-cn8l8"] Jul 16 21:12:58.261959 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.261893 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65dbbdbc54-cn8l8" podUID="8bcf6073-6a76-49c5-9133-0edc8f99578e" containerName="console" containerID="cri-o://2be77c6f111ad8a68df78a172656bbf67ac687931cafc43c2a5ffcb8f0adcce3" gracePeriod=15 Jul 16 21:12:58.502860 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.502835 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:12:58.605653 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.605579 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2hhs\" (UniqueName: \"kubernetes.io/projected/8bcf6073-6a76-49c5-9133-0edc8f99578e-kube-api-access-m2hhs\") pod \"8bcf6073-6a76-49c5-9133-0edc8f99578e\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " Jul 16 21:12:58.605653 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.605616 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-service-ca\") pod \"8bcf6073-6a76-49c5-9133-0edc8f99578e\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " Jul 16 21:12:58.605653 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.605634 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-config\") pod \"8bcf6073-6a76-49c5-9133-0edc8f99578e\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " Jul 16 21:12:58.605855 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.605657 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-oauth-config\") pod \"8bcf6073-6a76-49c5-9133-0edc8f99578e\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " Jul 16 21:12:58.605855 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.605682 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-oauth-serving-cert\") pod \"8bcf6073-6a76-49c5-9133-0edc8f99578e\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " Jul 16 21:12:58.605855 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.605752 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-serving-cert\") pod \"8bcf6073-6a76-49c5-9133-0edc8f99578e\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " Jul 16 21:12:58.605855 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.605835 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-trusted-ca-bundle\") pod \"8bcf6073-6a76-49c5-9133-0edc8f99578e\" (UID: \"8bcf6073-6a76-49c5-9133-0edc8f99578e\") " Jul 16 21:12:58.606102 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.606004 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-service-ca" (OuterVolumeSpecName: "service-ca") pod "8bcf6073-6a76-49c5-9133-0edc8f99578e" (UID: "8bcf6073-6a76-49c5-9133-0edc8f99578e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:12:58.606102 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.606080 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8bcf6073-6a76-49c5-9133-0edc8f99578e" (UID: "8bcf6073-6a76-49c5-9133-0edc8f99578e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:12:58.606102 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.606086 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-config" (OuterVolumeSpecName: "console-config") pod "8bcf6073-6a76-49c5-9133-0edc8f99578e" (UID: "8bcf6073-6a76-49c5-9133-0edc8f99578e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:12:58.606304 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.606285 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:58.606351 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.606312 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-config\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:58.606351 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.606327 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-oauth-serving-cert\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:58.606434 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.606364 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8bcf6073-6a76-49c5-9133-0edc8f99578e" (UID: "8bcf6073-6a76-49c5-9133-0edc8f99578e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:12:58.607931 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.607907 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8bcf6073-6a76-49c5-9133-0edc8f99578e" (UID: "8bcf6073-6a76-49c5-9133-0edc8f99578e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:12:58.608261 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.608235 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcf6073-6a76-49c5-9133-0edc8f99578e-kube-api-access-m2hhs" (OuterVolumeSpecName: "kube-api-access-m2hhs") pod "8bcf6073-6a76-49c5-9133-0edc8f99578e" (UID: "8bcf6073-6a76-49c5-9133-0edc8f99578e"). InnerVolumeSpecName "kube-api-access-m2hhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:12:58.608261 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.608238 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8bcf6073-6a76-49c5-9133-0edc8f99578e" (UID: "8bcf6073-6a76-49c5-9133-0edc8f99578e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:12:58.707248 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.707224 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2hhs\" (UniqueName: \"kubernetes.io/projected/8bcf6073-6a76-49c5-9133-0edc8f99578e-kube-api-access-m2hhs\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:58.707248 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.707244 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-oauth-config\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:58.707248 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.707254 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bcf6073-6a76-49c5-9133-0edc8f99578e-console-serving-cert\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:58.707434 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:58.707262 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bcf6073-6a76-49c5-9133-0edc8f99578e-trusted-ca-bundle\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:12:59.288033 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:59.288001 2577 generic.go:356] "Generic (PLEG): container finished" podID="8bcf6073-6a76-49c5-9133-0edc8f99578e" containerID="2be77c6f111ad8a68df78a172656bbf67ac687931cafc43c2a5ffcb8f0adcce3" exitCode=0 Jul 16 21:12:59.288457 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:59.288045 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dbbdbc54-cn8l8" Jul 16 21:12:59.288457 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:59.288051 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dbbdbc54-cn8l8" event={"ID":"8bcf6073-6a76-49c5-9133-0edc8f99578e","Type":"ContainerDied","Data":"2be77c6f111ad8a68df78a172656bbf67ac687931cafc43c2a5ffcb8f0adcce3"} Jul 16 21:12:59.288457 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:59.288089 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dbbdbc54-cn8l8" event={"ID":"8bcf6073-6a76-49c5-9133-0edc8f99578e","Type":"ContainerDied","Data":"189c85e08fae3ead2bd8cac5638a9ff9c3b7f6d33bfd714b4f2c630ad3ce8e8f"} Jul 16 21:12:59.288457 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:59.288111 2577 scope.go:117] "RemoveContainer" containerID="2be77c6f111ad8a68df78a172656bbf67ac687931cafc43c2a5ffcb8f0adcce3" Jul 16 21:12:59.296221 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:59.296178 2577 scope.go:117] "RemoveContainer" containerID="2be77c6f111ad8a68df78a172656bbf67ac687931cafc43c2a5ffcb8f0adcce3" Jul 16 21:12:59.296466 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:12:59.296447 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be77c6f111ad8a68df78a172656bbf67ac687931cafc43c2a5ffcb8f0adcce3\": container with ID starting with 2be77c6f111ad8a68df78a172656bbf67ac687931cafc43c2a5ffcb8f0adcce3 not found: ID does not exist" containerID="2be77c6f111ad8a68df78a172656bbf67ac687931cafc43c2a5ffcb8f0adcce3" Jul 16 21:12:59.296513 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:59.296475 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be77c6f111ad8a68df78a172656bbf67ac687931cafc43c2a5ffcb8f0adcce3"} err="failed to get container status \"2be77c6f111ad8a68df78a172656bbf67ac687931cafc43c2a5ffcb8f0adcce3\": rpc error: code = NotFound desc = could not find container \"2be77c6f111ad8a68df78a172656bbf67ac687931cafc43c2a5ffcb8f0adcce3\": container with ID starting with 2be77c6f111ad8a68df78a172656bbf67ac687931cafc43c2a5ffcb8f0adcce3 not found: ID does not exist" Jul 16 21:12:59.310957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:59.310937 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65dbbdbc54-cn8l8"] Jul 16 21:12:59.315691 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:12:59.315673 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65dbbdbc54-cn8l8"] Jul 16 21:13:00.449162 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:00.449130 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bcf6073-6a76-49c5-9133-0edc8f99578e" path="/var/lib/kubelet/pods/8bcf6073-6a76-49c5-9133-0edc8f99578e/volumes" Jul 16 21:13:38.485432 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.485395 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-546ddb9b4f-zxhzc"] Jul 16 21:13:38.485928 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.485711 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bcf6073-6a76-49c5-9133-0edc8f99578e" containerName="console" Jul 16 21:13:38.485928 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.485722 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bcf6073-6a76-49c5-9133-0edc8f99578e" containerName="console" Jul 16 21:13:38.485928 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.485778 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bcf6073-6a76-49c5-9133-0edc8f99578e" containerName="console" Jul 16 21:13:38.488532 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.488515 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.500568 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.500544 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546ddb9b4f-zxhzc"] Jul 16 21:13:38.572237 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.572207 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-trusted-ca-bundle\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.572400 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.572244 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqtn2\" (UniqueName: \"kubernetes.io/projected/92cbe129-7fcc-4674-956e-1930031c0ff2-kube-api-access-tqtn2\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.572400 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.572269 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92cbe129-7fcc-4674-956e-1930031c0ff2-console-oauth-config\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.572400 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.572291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-console-config\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.572400 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.572379 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-oauth-serving-cert\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.572603 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.572421 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92cbe129-7fcc-4674-956e-1930031c0ff2-console-serving-cert\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.572603 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.572484 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-service-ca\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.673675 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.673642 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-trusted-ca-bundle\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.673819 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.673678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqtn2\" (UniqueName: \"kubernetes.io/projected/92cbe129-7fcc-4674-956e-1930031c0ff2-kube-api-access-tqtn2\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.673819 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.673708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92cbe129-7fcc-4674-956e-1930031c0ff2-console-oauth-config\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.673819 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.673728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-console-config\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.673819 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.673757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-oauth-serving-cert\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.673819 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.673781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92cbe129-7fcc-4674-956e-1930031c0ff2-console-serving-cert\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.674050 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.673823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-service-ca\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.674622 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.674595 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-console-config\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.674738 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.674595 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-oauth-serving-cert\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.674738 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.674683 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-service-ca\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.674931 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.674912 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-trusted-ca-bundle\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.676331 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.676311 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92cbe129-7fcc-4674-956e-1930031c0ff2-console-oauth-config\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.676386 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.676312 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92cbe129-7fcc-4674-956e-1930031c0ff2-console-serving-cert\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.683150 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.683129 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqtn2\" (UniqueName: \"kubernetes.io/projected/92cbe129-7fcc-4674-956e-1930031c0ff2-kube-api-access-tqtn2\") pod \"console-546ddb9b4f-zxhzc\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.797080 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.797025 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:38.922956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:38.922923 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546ddb9b4f-zxhzc"] Jul 16 21:13:38.925298 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:13:38.925275 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cbe129_7fcc_4674_956e_1930031c0ff2.slice/crio-cdde7e7fe040003e2a4f34371d68883faa4c62d65617d1d02980eff8f2412da8 WatchSource:0}: Error finding container cdde7e7fe040003e2a4f34371d68883faa4c62d65617d1d02980eff8f2412da8: Status 404 returned error can't find the container with id cdde7e7fe040003e2a4f34371d68883faa4c62d65617d1d02980eff8f2412da8 Jul 16 21:13:39.434071 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:39.434043 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546ddb9b4f-zxhzc" event={"ID":"92cbe129-7fcc-4674-956e-1930031c0ff2","Type":"ContainerStarted","Data":"cfad0f14de7952ad7e4eaeed3f321defa77ffe2abad64c10334d844b8858087a"} Jul 16 21:13:39.434246 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:39.434091 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546ddb9b4f-zxhzc" event={"ID":"92cbe129-7fcc-4674-956e-1930031c0ff2","Type":"ContainerStarted","Data":"cdde7e7fe040003e2a4f34371d68883faa4c62d65617d1d02980eff8f2412da8"} Jul 16 21:13:39.454267 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:39.454221 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-546ddb9b4f-zxhzc" podStartSLOduration=1.454206041 podStartE2EDuration="1.454206041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:13:39.45300246 +0000 UTC m=+235.628110679" watchObservedRunningTime="2026-07-16 21:13:39.454206041 +0000 UTC m=+235.629314225" Jul 16 21:13:48.797667 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:48.797627 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:48.798143 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:48.797700 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:48.802561 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:48.802537 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:49.473858 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:49.473829 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:13:49.525239 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:13:49.525208 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-574f89c6f4-ck4mn"] Jul 16 21:14:14.550902 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.550790 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-574f89c6f4-ck4mn" podUID="9f34028c-6657-405d-966a-1f9292185a33" containerName="console" containerID="cri-o://20e104f3151bd482dd2faa20a935da21b7edf5e6fee909387656f48f92ba8091" gracePeriod=15 Jul 16 21:14:14.788464 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.788441 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:14:14.849849 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.849757 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-oauth-serving-cert\") pod \"9f34028c-6657-405d-966a-1f9292185a33\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " Jul 16 21:14:14.849849 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.849803 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8jb9\" (UniqueName: \"kubernetes.io/projected/9f34028c-6657-405d-966a-1f9292185a33-kube-api-access-m8jb9\") pod \"9f34028c-6657-405d-966a-1f9292185a33\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " Jul 16 21:14:14.849849 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.849848 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-service-ca\") pod \"9f34028c-6657-405d-966a-1f9292185a33\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " Jul 16 21:14:14.850111 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.849883 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-console-config\") pod \"9f34028c-6657-405d-966a-1f9292185a33\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " Jul 16 21:14:14.850111 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.849930 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-trusted-ca-bundle\") pod \"9f34028c-6657-405d-966a-1f9292185a33\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " Jul 16 21:14:14.850111 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.849985 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f34028c-6657-405d-966a-1f9292185a33-console-serving-cert\") pod \"9f34028c-6657-405d-966a-1f9292185a33\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " Jul 16 21:14:14.850111 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.850008 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f34028c-6657-405d-966a-1f9292185a33-console-oauth-config\") pod \"9f34028c-6657-405d-966a-1f9292185a33\" (UID: \"9f34028c-6657-405d-966a-1f9292185a33\") " Jul 16 21:14:14.850328 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.850277 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9f34028c-6657-405d-966a-1f9292185a33" (UID: "9f34028c-6657-405d-966a-1f9292185a33"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:14:14.850328 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.850315 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-service-ca" (OuterVolumeSpecName: "service-ca") pod "9f34028c-6657-405d-966a-1f9292185a33" (UID: "9f34028c-6657-405d-966a-1f9292185a33"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:14:14.850434 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.850320 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-console-config" (OuterVolumeSpecName: "console-config") pod "9f34028c-6657-405d-966a-1f9292185a33" (UID: "9f34028c-6657-405d-966a-1f9292185a33"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:14:14.850434 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.850383 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9f34028c-6657-405d-966a-1f9292185a33" (UID: "9f34028c-6657-405d-966a-1f9292185a33"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:14:14.852075 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.852043 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f34028c-6657-405d-966a-1f9292185a33-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9f34028c-6657-405d-966a-1f9292185a33" (UID: "9f34028c-6657-405d-966a-1f9292185a33"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:14:14.852075 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.852068 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f34028c-6657-405d-966a-1f9292185a33-kube-api-access-m8jb9" (OuterVolumeSpecName: "kube-api-access-m8jb9") pod "9f34028c-6657-405d-966a-1f9292185a33" (UID: "9f34028c-6657-405d-966a-1f9292185a33"). InnerVolumeSpecName "kube-api-access-m8jb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:14:14.852280 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.852125 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f34028c-6657-405d-966a-1f9292185a33-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9f34028c-6657-405d-966a-1f9292185a33" (UID: "9f34028c-6657-405d-966a-1f9292185a33"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:14:14.950653 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.950618 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-console-config\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:14:14.950653 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.950647 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-trusted-ca-bundle\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:14:14.950653 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.950660 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f34028c-6657-405d-966a-1f9292185a33-console-serving-cert\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:14:14.950886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.950672 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f34028c-6657-405d-966a-1f9292185a33-console-oauth-config\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:14:14.950886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.950684 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-oauth-serving-cert\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:14:14.950886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.950696 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8jb9\" (UniqueName: \"kubernetes.io/projected/9f34028c-6657-405d-966a-1f9292185a33-kube-api-access-m8jb9\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:14:14.950886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:14.950707 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f34028c-6657-405d-966a-1f9292185a33-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:14:15.564055 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:15.564019 2577 generic.go:356] "Generic (PLEG): container finished" podID="9f34028c-6657-405d-966a-1f9292185a33" containerID="20e104f3151bd482dd2faa20a935da21b7edf5e6fee909387656f48f92ba8091" exitCode=0 Jul 16 21:14:15.564055 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:15.564056 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574f89c6f4-ck4mn" event={"ID":"9f34028c-6657-405d-966a-1f9292185a33","Type":"ContainerDied","Data":"20e104f3151bd482dd2faa20a935da21b7edf5e6fee909387656f48f92ba8091"} Jul 16 21:14:15.564590 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:15.564066 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574f89c6f4-ck4mn" Jul 16 21:14:15.564590 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:15.564090 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574f89c6f4-ck4mn" event={"ID":"9f34028c-6657-405d-966a-1f9292185a33","Type":"ContainerDied","Data":"49e820dbb6cb67789a4398d479bf95fef1586f5cf0b3db443423ab10d866504d"} Jul 16 21:14:15.564590 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:15.564111 2577 scope.go:117] "RemoveContainer" containerID="20e104f3151bd482dd2faa20a935da21b7edf5e6fee909387656f48f92ba8091" Jul 16 21:14:15.572296 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:15.572278 2577 scope.go:117] "RemoveContainer" containerID="20e104f3151bd482dd2faa20a935da21b7edf5e6fee909387656f48f92ba8091" Jul 16 21:14:15.572541 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:14:15.572522 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e104f3151bd482dd2faa20a935da21b7edf5e6fee909387656f48f92ba8091\": container with ID starting with 20e104f3151bd482dd2faa20a935da21b7edf5e6fee909387656f48f92ba8091 not found: ID does not exist" containerID="20e104f3151bd482dd2faa20a935da21b7edf5e6fee909387656f48f92ba8091" Jul 16 21:14:15.572590 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:15.572554 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e104f3151bd482dd2faa20a935da21b7edf5e6fee909387656f48f92ba8091"} err="failed to get container status \"20e104f3151bd482dd2faa20a935da21b7edf5e6fee909387656f48f92ba8091\": rpc error: code = NotFound desc = could not find container \"20e104f3151bd482dd2faa20a935da21b7edf5e6fee909387656f48f92ba8091\": container with ID starting with 20e104f3151bd482dd2faa20a935da21b7edf5e6fee909387656f48f92ba8091 not found: ID does not exist" Jul 16 21:14:15.584989 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:15.584970 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-574f89c6f4-ck4mn"] Jul 16 21:14:15.591118 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:15.591096 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-574f89c6f4-ck4mn"] Jul 16 21:14:16.449581 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:16.449551 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f34028c-6657-405d-966a-1f9292185a33" path="/var/lib/kubelet/pods/9f34028c-6657-405d-966a-1f9292185a33/volumes" Jul 16 21:14:44.223478 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:44.223448 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:14:44.228605 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:44.228586 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:14:44.228801 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:14:44.228785 2577 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 21:15:00.663508 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.663475 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh"] Jul 16 21:15:00.665957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.664127 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f34028c-6657-405d-966a-1f9292185a33" containerName="console" Jul 16 21:15:00.665957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.664149 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f34028c-6657-405d-966a-1f9292185a33" containerName="console" Jul 16 21:15:00.665957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.664418 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f34028c-6657-405d-966a-1f9292185a33" containerName="console" Jul 16 21:15:00.670343 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.670322 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" Jul 16 21:15:00.673261 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.673239 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 21:15:00.674298 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.674283 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 21:15:00.674351 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.674306 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-thrxk\"" Jul 16 21:15:00.678806 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.678787 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh"] Jul 16 21:15:00.766236 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.766213 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hf7b\" (UniqueName: \"kubernetes.io/projected/f2b65d6f-3899-45fc-91e9-075a9241d8ca-kube-api-access-6hf7b\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh\" (UID: \"f2b65d6f-3899-45fc-91e9-075a9241d8ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" Jul 16 21:15:00.766371 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.766253 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2b65d6f-3899-45fc-91e9-075a9241d8ca-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh\" (UID: \"f2b65d6f-3899-45fc-91e9-075a9241d8ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" Jul 16 21:15:00.766371 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.766289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2b65d6f-3899-45fc-91e9-075a9241d8ca-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh\" (UID: \"f2b65d6f-3899-45fc-91e9-075a9241d8ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" Jul 16 21:15:00.866899 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.866874 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hf7b\" (UniqueName: \"kubernetes.io/projected/f2b65d6f-3899-45fc-91e9-075a9241d8ca-kube-api-access-6hf7b\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh\" (UID: \"f2b65d6f-3899-45fc-91e9-075a9241d8ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" Jul 16 21:15:00.867002 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.866914 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2b65d6f-3899-45fc-91e9-075a9241d8ca-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh\" (UID: \"f2b65d6f-3899-45fc-91e9-075a9241d8ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" Jul 16 21:15:00.867075 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.867050 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2b65d6f-3899-45fc-91e9-075a9241d8ca-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh\" (UID: \"f2b65d6f-3899-45fc-91e9-075a9241d8ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" Jul 16 21:15:00.867302 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.867284 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2b65d6f-3899-45fc-91e9-075a9241d8ca-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh\" (UID: \"f2b65d6f-3899-45fc-91e9-075a9241d8ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" Jul 16 21:15:00.867350 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.867343 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2b65d6f-3899-45fc-91e9-075a9241d8ca-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh\" (UID: \"f2b65d6f-3899-45fc-91e9-075a9241d8ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" Jul 16 21:15:00.876436 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.876418 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hf7b\" (UniqueName: \"kubernetes.io/projected/f2b65d6f-3899-45fc-91e9-075a9241d8ca-kube-api-access-6hf7b\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh\" (UID: \"f2b65d6f-3899-45fc-91e9-075a9241d8ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" Jul 16 21:15:00.980061 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:00.980009 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" Jul 16 21:15:01.117216 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:01.117173 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh"] Jul 16 21:15:01.120930 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:15:01.120903 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b65d6f_3899_45fc_91e9_075a9241d8ca.slice/crio-4cf422bf76f1affef06db2e2ee4bbccd23bad4347d9d5928a61a8fa66fa4492a WatchSource:0}: Error finding container 4cf422bf76f1affef06db2e2ee4bbccd23bad4347d9d5928a61a8fa66fa4492a: Status 404 returned error can't find the container with id 4cf422bf76f1affef06db2e2ee4bbccd23bad4347d9d5928a61a8fa66fa4492a Jul 16 21:15:01.123175 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:01.123157 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:15:01.730560 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:01.730532 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" event={"ID":"f2b65d6f-3899-45fc-91e9-075a9241d8ca","Type":"ContainerStarted","Data":"4cf422bf76f1affef06db2e2ee4bbccd23bad4347d9d5928a61a8fa66fa4492a"} Jul 16 21:15:09.763984 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:09.763953 2577 generic.go:356] "Generic (PLEG): container finished" podID="f2b65d6f-3899-45fc-91e9-075a9241d8ca" containerID="f4b623224b77ee72109f1695e327decf987de617870308d634a241f4621cb92d" exitCode=0 Jul 16 21:15:09.764412 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:09.764052 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" event={"ID":"f2b65d6f-3899-45fc-91e9-075a9241d8ca","Type":"ContainerDied","Data":"f4b623224b77ee72109f1695e327decf987de617870308d634a241f4621cb92d"} Jul 16 21:15:14.783776 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:14.783737 2577 generic.go:356] "Generic (PLEG): container finished" podID="f2b65d6f-3899-45fc-91e9-075a9241d8ca" containerID="ba11a89178df811e7ad0d70659b7c6882b63a7e5cab8f56f903131af42a3d9e7" exitCode=0 Jul 16 21:15:14.784152 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:14.783827 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" event={"ID":"f2b65d6f-3899-45fc-91e9-075a9241d8ca","Type":"ContainerDied","Data":"ba11a89178df811e7ad0d70659b7c6882b63a7e5cab8f56f903131af42a3d9e7"} Jul 16 21:15:23.823893 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:23.823855 2577 generic.go:356] "Generic (PLEG): container finished" podID="f2b65d6f-3899-45fc-91e9-075a9241d8ca" containerID="a3c597eea314ef4c2037f9e84e1e73214216d3a9186515f9f8d1928d16f5946d" exitCode=0 Jul 16 21:15:23.824292 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:23.823939 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" event={"ID":"f2b65d6f-3899-45fc-91e9-075a9241d8ca","Type":"ContainerDied","Data":"a3c597eea314ef4c2037f9e84e1e73214216d3a9186515f9f8d1928d16f5946d"} Jul 16 21:15:24.974388 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:24.974360 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" Jul 16 21:15:25.078421 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:25.078338 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2b65d6f-3899-45fc-91e9-075a9241d8ca-util\") pod \"f2b65d6f-3899-45fc-91e9-075a9241d8ca\" (UID: \"f2b65d6f-3899-45fc-91e9-075a9241d8ca\") " Jul 16 21:15:25.078584 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:25.078465 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hf7b\" (UniqueName: \"kubernetes.io/projected/f2b65d6f-3899-45fc-91e9-075a9241d8ca-kube-api-access-6hf7b\") pod \"f2b65d6f-3899-45fc-91e9-075a9241d8ca\" (UID: \"f2b65d6f-3899-45fc-91e9-075a9241d8ca\") " Jul 16 21:15:25.078584 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:25.078490 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2b65d6f-3899-45fc-91e9-075a9241d8ca-bundle\") pod \"f2b65d6f-3899-45fc-91e9-075a9241d8ca\" (UID: \"f2b65d6f-3899-45fc-91e9-075a9241d8ca\") " Jul 16 21:15:25.079241 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:25.079201 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b65d6f-3899-45fc-91e9-075a9241d8ca-bundle" (OuterVolumeSpecName: "bundle") pod "f2b65d6f-3899-45fc-91e9-075a9241d8ca" (UID: "f2b65d6f-3899-45fc-91e9-075a9241d8ca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:15:25.080671 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:25.080645 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b65d6f-3899-45fc-91e9-075a9241d8ca-kube-api-access-6hf7b" (OuterVolumeSpecName: "kube-api-access-6hf7b") pod "f2b65d6f-3899-45fc-91e9-075a9241d8ca" (UID: "f2b65d6f-3899-45fc-91e9-075a9241d8ca"). InnerVolumeSpecName "kube-api-access-6hf7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:15:25.082601 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:25.082583 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b65d6f-3899-45fc-91e9-075a9241d8ca-util" (OuterVolumeSpecName: "util") pod "f2b65d6f-3899-45fc-91e9-075a9241d8ca" (UID: "f2b65d6f-3899-45fc-91e9-075a9241d8ca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:15:25.179159 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:25.179130 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2b65d6f-3899-45fc-91e9-075a9241d8ca-util\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:15:25.179159 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:25.179154 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hf7b\" (UniqueName: \"kubernetes.io/projected/f2b65d6f-3899-45fc-91e9-075a9241d8ca-kube-api-access-6hf7b\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:15:25.179159 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:25.179164 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2b65d6f-3899-45fc-91e9-075a9241d8ca-bundle\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:15:25.833360 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:25.833327 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" event={"ID":"f2b65d6f-3899-45fc-91e9-075a9241d8ca","Type":"ContainerDied","Data":"4cf422bf76f1affef06db2e2ee4bbccd23bad4347d9d5928a61a8fa66fa4492a"} Jul 16 21:15:25.833360 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:25.833356 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cmcfh" Jul 16 21:15:25.833556 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:25.833375 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cf422bf76f1affef06db2e2ee4bbccd23bad4347d9d5928a61a8fa66fa4492a" Jul 16 21:15:30.115534 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.115493 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nqqgn"] Jul 16 21:15:30.116094 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.115892 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b65d6f-3899-45fc-91e9-075a9241d8ca" containerName="pull" Jul 16 21:15:30.116094 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.115910 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b65d6f-3899-45fc-91e9-075a9241d8ca" containerName="pull" Jul 16 21:15:30.116094 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.115930 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b65d6f-3899-45fc-91e9-075a9241d8ca" containerName="extract" Jul 16 21:15:30.116094 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.115939 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b65d6f-3899-45fc-91e9-075a9241d8ca" containerName="extract" Jul 16 21:15:30.116094 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.115955 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b65d6f-3899-45fc-91e9-075a9241d8ca" containerName="util" Jul 16 21:15:30.116094 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.115964 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b65d6f-3899-45fc-91e9-075a9241d8ca" containerName="util" Jul 16 21:15:30.116094 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.116041 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b65d6f-3899-45fc-91e9-075a9241d8ca" containerName="extract" Jul 16 21:15:30.118373 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.118354 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nqqgn" Jul 16 21:15:30.125053 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.125034 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 16 21:15:30.125151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.125041 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:15:30.125151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.125101 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8wlzc\"" Jul 16 21:15:30.177872 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.177841 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nqqgn"] Jul 16 21:15:30.221666 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.221627 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxkcg\" (UniqueName: \"kubernetes.io/projected/ffcf3bd5-acac-461f-9900-bcff72183307-kube-api-access-xxkcg\") pod \"cert-manager-operator-controller-manager-55c444bf86-nqqgn\" (UID: \"ffcf3bd5-acac-461f-9900-bcff72183307\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nqqgn" Jul 16 21:15:30.221828 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.221676 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffcf3bd5-acac-461f-9900-bcff72183307-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nqqgn\" (UID: \"ffcf3bd5-acac-461f-9900-bcff72183307\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nqqgn" Jul 16 21:15:30.323161 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.323127 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxkcg\" (UniqueName: \"kubernetes.io/projected/ffcf3bd5-acac-461f-9900-bcff72183307-kube-api-access-xxkcg\") pod \"cert-manager-operator-controller-manager-55c444bf86-nqqgn\" (UID: \"ffcf3bd5-acac-461f-9900-bcff72183307\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nqqgn" Jul 16 21:15:30.323344 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.323170 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffcf3bd5-acac-461f-9900-bcff72183307-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nqqgn\" (UID: \"ffcf3bd5-acac-461f-9900-bcff72183307\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nqqgn" Jul 16 21:15:30.323533 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.323516 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffcf3bd5-acac-461f-9900-bcff72183307-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nqqgn\" (UID: \"ffcf3bd5-acac-461f-9900-bcff72183307\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nqqgn" Jul 16 21:15:30.336464 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.336439 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxkcg\" (UniqueName: \"kubernetes.io/projected/ffcf3bd5-acac-461f-9900-bcff72183307-kube-api-access-xxkcg\") pod \"cert-manager-operator-controller-manager-55c444bf86-nqqgn\" (UID: \"ffcf3bd5-acac-461f-9900-bcff72183307\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nqqgn" Jul 16 21:15:30.427425 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.427395 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nqqgn" Jul 16 21:15:30.587644 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.587618 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nqqgn"] Jul 16 21:15:30.590019 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:15:30.589992 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcf3bd5_acac_461f_9900_bcff72183307.slice/crio-3670bf5002f28b401092849f9c4839a6cf3fb096b0076d0971c66a61808e9f89 WatchSource:0}: Error finding container 3670bf5002f28b401092849f9c4839a6cf3fb096b0076d0971c66a61808e9f89: Status 404 returned error can't find the container with id 3670bf5002f28b401092849f9c4839a6cf3fb096b0076d0971c66a61808e9f89 Jul 16 21:15:30.854478 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:30.854366 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nqqgn" event={"ID":"ffcf3bd5-acac-461f-9900-bcff72183307","Type":"ContainerStarted","Data":"3670bf5002f28b401092849f9c4839a6cf3fb096b0076d0971c66a61808e9f89"} Jul 16 21:15:33.874980 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:33.874940 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nqqgn" event={"ID":"ffcf3bd5-acac-461f-9900-bcff72183307","Type":"ContainerStarted","Data":"93005e865451bea55f0e5febcfa2b63e72136516be14004db5cfccf0c27bd7e0"} Jul 16 21:15:34.404278 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:34.404223 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nqqgn" podStartSLOduration=2.144374148 podStartE2EDuration="4.404207957s" totalImagesPullingTime="2.259833809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:15:34.013600763 +0000 UTC m=+350.188708955" watchObservedRunningTime="2026-07-16 21:15:34.404207957 +0000 UTC m=+350.579316148" Jul 16 21:15:34.404935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:34.404914 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mjrws"] Jul 16 21:15:34.407351 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:34.407330 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mjrws" Jul 16 21:15:34.414073 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:34.414054 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 16 21:15:34.414176 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:34.414095 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 16 21:15:34.455043 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:34.455013 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mjrws"] Jul 16 21:15:34.557588 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:34.557558 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/332f4b30-e251-48b5-8907-b698325b3e46-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mjrws\" (UID: \"332f4b30-e251-48b5-8907-b698325b3e46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mjrws" Jul 16 21:15:34.557757 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:34.557617 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q6xj\" (UniqueName: \"kubernetes.io/projected/332f4b30-e251-48b5-8907-b698325b3e46-kube-api-access-2q6xj\") pod \"cert-manager-webhook-6486ffd997-mjrws\" (UID: \"332f4b30-e251-48b5-8907-b698325b3e46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mjrws" Jul 16 21:15:34.659027 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:34.658936 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q6xj\" (UniqueName: \"kubernetes.io/projected/332f4b30-e251-48b5-8907-b698325b3e46-kube-api-access-2q6xj\") pod \"cert-manager-webhook-6486ffd997-mjrws\" (UID: \"332f4b30-e251-48b5-8907-b698325b3e46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mjrws" Jul 16 21:15:34.659027 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:34.659010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/332f4b30-e251-48b5-8907-b698325b3e46-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mjrws\" (UID: \"332f4b30-e251-48b5-8907-b698325b3e46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mjrws" Jul 16 21:15:34.685743 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:34.685711 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/332f4b30-e251-48b5-8907-b698325b3e46-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mjrws\" (UID: \"332f4b30-e251-48b5-8907-b698325b3e46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mjrws" Jul 16 21:15:34.686352 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:34.686336 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q6xj\" (UniqueName: \"kubernetes.io/projected/332f4b30-e251-48b5-8907-b698325b3e46-kube-api-access-2q6xj\") pod \"cert-manager-webhook-6486ffd997-mjrws\" (UID: \"332f4b30-e251-48b5-8907-b698325b3e46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mjrws" Jul 16 21:15:34.727083 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:34.727056 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mjrws" Jul 16 21:15:34.902325 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:34.902301 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mjrws"] Jul 16 21:15:34.903332 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:15:34.903307 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod332f4b30_e251_48b5_8907_b698325b3e46.slice/crio-c401966202f0d63205e34b755a8dfb369e3dbd6fcc8afd614ed40c9488b1e9ad WatchSource:0}: Error finding container c401966202f0d63205e34b755a8dfb369e3dbd6fcc8afd614ed40c9488b1e9ad: Status 404 returned error can't find the container with id c401966202f0d63205e34b755a8dfb369e3dbd6fcc8afd614ed40c9488b1e9ad Jul 16 21:15:35.885080 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:35.884992 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mjrws" event={"ID":"332f4b30-e251-48b5-8907-b698325b3e46","Type":"ContainerStarted","Data":"c401966202f0d63205e34b755a8dfb369e3dbd6fcc8afd614ed40c9488b1e9ad"} Jul 16 21:15:35.907560 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:35.907527 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v"] Jul 16 21:15:35.910051 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:35.910025 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" Jul 16 21:15:35.915948 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:15:35.915899 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v\" is forbidden: User \"system:node:ip-10-0-137-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-137-133.ec2.internal' and this object" podUID="3bb96891-a061-4969-9368-ad791fbaba28" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" Jul 16 21:15:35.916100 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:35.916007 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 21:15:35.916944 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:35.916923 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-thrxk\"" Jul 16 21:15:35.917063 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:35.916961 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 21:15:35.948811 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:35.948780 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v"] Jul 16 21:15:36.071360 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:36.071323 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bb96891-a061-4969-9368-ad791fbaba28-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v\" (UID: \"3bb96891-a061-4969-9368-ad791fbaba28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" Jul 16 21:15:36.071546 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:36.071392 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tmgf\" (UniqueName: \"kubernetes.io/projected/3bb96891-a061-4969-9368-ad791fbaba28-kube-api-access-6tmgf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v\" (UID: \"3bb96891-a061-4969-9368-ad791fbaba28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" Jul 16 21:15:36.071546 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:36.071502 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bb96891-a061-4969-9368-ad791fbaba28-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v\" (UID: \"3bb96891-a061-4969-9368-ad791fbaba28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" Jul 16 21:15:36.172445 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:36.172360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bb96891-a061-4969-9368-ad791fbaba28-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v\" (UID: \"3bb96891-a061-4969-9368-ad791fbaba28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" Jul 16 21:15:36.172445 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:36.172429 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tmgf\" (UniqueName: \"kubernetes.io/projected/3bb96891-a061-4969-9368-ad791fbaba28-kube-api-access-6tmgf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v\" (UID: \"3bb96891-a061-4969-9368-ad791fbaba28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" Jul 16 21:15:36.172653 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:36.172498 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bb96891-a061-4969-9368-ad791fbaba28-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v\" (UID: \"3bb96891-a061-4969-9368-ad791fbaba28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" Jul 16 21:15:36.172844 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:36.172821 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bb96891-a061-4969-9368-ad791fbaba28-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v\" (UID: \"3bb96891-a061-4969-9368-ad791fbaba28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" Jul 16 21:15:36.172892 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:36.172867 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bb96891-a061-4969-9368-ad791fbaba28-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v\" (UID: \"3bb96891-a061-4969-9368-ad791fbaba28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" Jul 16 21:15:36.199803 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:36.199780 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tmgf\" (UniqueName: \"kubernetes.io/projected/3bb96891-a061-4969-9368-ad791fbaba28-kube-api-access-6tmgf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v\" (UID: \"3bb96891-a061-4969-9368-ad791fbaba28\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" Jul 16 21:15:36.220698 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:36.220671 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" Jul 16 21:15:36.407623 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:36.407539 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v"] Jul 16 21:15:36.410834 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:15:36.410804 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb96891_a061_4969_9368_ad791fbaba28.slice/crio-2d12fc5b1ebe95ec2fb64600e55f9e22ac3c65f87460b3135dfe5790e620e319 WatchSource:0}: Error finding container 2d12fc5b1ebe95ec2fb64600e55f9e22ac3c65f87460b3135dfe5790e620e319: Status 404 returned error can't find the container with id 2d12fc5b1ebe95ec2fb64600e55f9e22ac3c65f87460b3135dfe5790e620e319 Jul 16 21:15:36.891346 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:36.891304 2577 generic.go:356] "Generic (PLEG): container finished" podID="3bb96891-a061-4969-9368-ad791fbaba28" containerID="8ceea457008bf2f8efd59fd78cfac1d1cea19af243bbf895e9d80653915ab812" exitCode=0 Jul 16 21:15:36.891546 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:36.891401 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" event={"ID":"3bb96891-a061-4969-9368-ad791fbaba28","Type":"ContainerDied","Data":"8ceea457008bf2f8efd59fd78cfac1d1cea19af243bbf895e9d80653915ab812"} Jul 16 21:15:36.891546 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:36.891466 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" event={"ID":"3bb96891-a061-4969-9368-ad791fbaba28","Type":"ContainerStarted","Data":"2d12fc5b1ebe95ec2fb64600e55f9e22ac3c65f87460b3135dfe5790e620e319"} Jul 16 21:15:37.898285 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:37.898254 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mjrws" event={"ID":"332f4b30-e251-48b5-8907-b698325b3e46","Type":"ContainerStarted","Data":"07acc6d51943a3af3bcfb40fb3d259a7aa53c30cfcb3373a7edea89f40d29e7d"} Jul 16 21:15:37.898669 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:37.898359 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mjrws" Jul 16 21:15:37.979897 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:37.979844 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-mjrws" podStartSLOduration=1.393764215 podStartE2EDuration="3.979828938s" totalImagesPullingTime="2.586064723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:15:37.978572848 +0000 UTC m=+354.153681051" watchObservedRunningTime="2026-07-16 21:15:37.979828938 +0000 UTC m=+354.154937129" Jul 16 21:15:39.510267 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:39.510234 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zrhf6"] Jul 16 21:15:39.512774 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:39.512759 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zrhf6" Jul 16 21:15:39.518267 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:39.518241 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-k2b2h\"" Jul 16 21:15:39.556513 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:39.556443 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zrhf6"] Jul 16 21:15:39.606225 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:39.606195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24480ade-c1c9-4465-949a-c32dd2fec851-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zrhf6\" (UID: \"24480ade-c1c9-4465-949a-c32dd2fec851\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zrhf6" Jul 16 21:15:39.606361 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:39.606249 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg6w6\" (UniqueName: \"kubernetes.io/projected/24480ade-c1c9-4465-949a-c32dd2fec851-kube-api-access-lg6w6\") pod \"cert-manager-cainjector-559cdd8c87-zrhf6\" (UID: \"24480ade-c1c9-4465-949a-c32dd2fec851\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zrhf6" Jul 16 21:15:39.707084 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:39.707047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24480ade-c1c9-4465-949a-c32dd2fec851-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zrhf6\" (UID: \"24480ade-c1c9-4465-949a-c32dd2fec851\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zrhf6" Jul 16 21:15:39.707262 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:39.707097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg6w6\" (UniqueName: \"kubernetes.io/projected/24480ade-c1c9-4465-949a-c32dd2fec851-kube-api-access-lg6w6\") pod \"cert-manager-cainjector-559cdd8c87-zrhf6\" (UID: \"24480ade-c1c9-4465-949a-c32dd2fec851\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zrhf6" Jul 16 21:15:39.731133 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:39.731097 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg6w6\" (UniqueName: \"kubernetes.io/projected/24480ade-c1c9-4465-949a-c32dd2fec851-kube-api-access-lg6w6\") pod \"cert-manager-cainjector-559cdd8c87-zrhf6\" (UID: \"24480ade-c1c9-4465-949a-c32dd2fec851\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zrhf6" Jul 16 21:15:39.779107 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:39.779075 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24480ade-c1c9-4465-949a-c32dd2fec851-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zrhf6\" (UID: \"24480ade-c1c9-4465-949a-c32dd2fec851\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zrhf6" Jul 16 21:15:39.821408 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:39.821327 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zrhf6" Jul 16 21:15:39.911202 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:39.911157 2577 generic.go:356] "Generic (PLEG): container finished" podID="3bb96891-a061-4969-9368-ad791fbaba28" containerID="072e3614991455853ff52d15b81487d807c82046e0e623a98a700519e803fba1" exitCode=0 Jul 16 21:15:39.911327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:39.911221 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" event={"ID":"3bb96891-a061-4969-9368-ad791fbaba28","Type":"ContainerDied","Data":"072e3614991455853ff52d15b81487d807c82046e0e623a98a700519e803fba1"} Jul 16 21:15:39.996510 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:39.996444 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zrhf6"] Jul 16 21:15:40.000117 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:15:40.000090 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24480ade_c1c9_4465_949a_c32dd2fec851.slice/crio-c6555c419e9c320f58ca46717ca1e575eca76f50e04d14e256a60ba136729ba5 WatchSource:0}: Error finding container c6555c419e9c320f58ca46717ca1e575eca76f50e04d14e256a60ba136729ba5: Status 404 returned error can't find the container with id c6555c419e9c320f58ca46717ca1e575eca76f50e04d14e256a60ba136729ba5 Jul 16 21:15:40.917270 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:40.917233 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zrhf6" event={"ID":"24480ade-c1c9-4465-949a-c32dd2fec851","Type":"ContainerStarted","Data":"3d1fa49d9348af109bdcf0a336b1cdd143a1f4a9b794f57341f7dbce7b8a31f6"} Jul 16 21:15:40.917802 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:40.917283 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zrhf6" event={"ID":"24480ade-c1c9-4465-949a-c32dd2fec851","Type":"ContainerStarted","Data":"c6555c419e9c320f58ca46717ca1e575eca76f50e04d14e256a60ba136729ba5"} Jul 16 21:15:40.919368 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:40.919343 2577 generic.go:356] "Generic (PLEG): container finished" podID="3bb96891-a061-4969-9368-ad791fbaba28" containerID="54ad4a26796535fbdf64e049a2cf8660efd08e37054cf1a6ac3ad82e537da5f7" exitCode=0 Jul 16 21:15:40.919475 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:40.919384 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" event={"ID":"3bb96891-a061-4969-9368-ad791fbaba28","Type":"ContainerDied","Data":"54ad4a26796535fbdf64e049a2cf8660efd08e37054cf1a6ac3ad82e537da5f7"} Jul 16 21:15:40.991056 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:40.990983 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zrhf6" podStartSLOduration=1.990960314 podStartE2EDuration="1.990960314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:15:40.949323594 +0000 UTC m=+357.124431790" watchObservedRunningTime="2026-07-16 21:15:40.990960314 +0000 UTC m=+357.166068507" Jul 16 21:15:42.044842 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:42.044821 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" Jul 16 21:15:42.227878 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:42.227798 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bb96891-a061-4969-9368-ad791fbaba28-bundle\") pod \"3bb96891-a061-4969-9368-ad791fbaba28\" (UID: \"3bb96891-a061-4969-9368-ad791fbaba28\") " Jul 16 21:15:42.227878 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:42.227859 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tmgf\" (UniqueName: \"kubernetes.io/projected/3bb96891-a061-4969-9368-ad791fbaba28-kube-api-access-6tmgf\") pod \"3bb96891-a061-4969-9368-ad791fbaba28\" (UID: \"3bb96891-a061-4969-9368-ad791fbaba28\") " Jul 16 21:15:42.227878 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:42.227875 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bb96891-a061-4969-9368-ad791fbaba28-util\") pod \"3bb96891-a061-4969-9368-ad791fbaba28\" (UID: \"3bb96891-a061-4969-9368-ad791fbaba28\") " Jul 16 21:15:42.228235 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:42.228213 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb96891-a061-4969-9368-ad791fbaba28-bundle" (OuterVolumeSpecName: "bundle") pod "3bb96891-a061-4969-9368-ad791fbaba28" (UID: "3bb96891-a061-4969-9368-ad791fbaba28"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:15:42.229928 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:42.229904 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb96891-a061-4969-9368-ad791fbaba28-kube-api-access-6tmgf" (OuterVolumeSpecName: "kube-api-access-6tmgf") pod "3bb96891-a061-4969-9368-ad791fbaba28" (UID: "3bb96891-a061-4969-9368-ad791fbaba28"). InnerVolumeSpecName "kube-api-access-6tmgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:15:42.231957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:42.231936 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb96891-a061-4969-9368-ad791fbaba28-util" (OuterVolumeSpecName: "util") pod "3bb96891-a061-4969-9368-ad791fbaba28" (UID: "3bb96891-a061-4969-9368-ad791fbaba28"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:15:42.328907 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:42.328868 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bb96891-a061-4969-9368-ad791fbaba28-bundle\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:15:42.328907 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:42.328902 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tmgf\" (UniqueName: \"kubernetes.io/projected/3bb96891-a061-4969-9368-ad791fbaba28-kube-api-access-6tmgf\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:15:42.329099 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:42.328916 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bb96891-a061-4969-9368-ad791fbaba28-util\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:15:42.929529 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:42.929496 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" event={"ID":"3bb96891-a061-4969-9368-ad791fbaba28","Type":"ContainerDied","Data":"2d12fc5b1ebe95ec2fb64600e55f9e22ac3c65f87460b3135dfe5790e620e319"} Jul 16 21:15:42.929687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:42.929540 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d12fc5b1ebe95ec2fb64600e55f9e22ac3c65f87460b3135dfe5790e620e319" Jul 16 21:15:42.929687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:42.929551 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmf59v" Jul 16 21:15:43.906087 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:43.906058 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mjrws" Jul 16 21:15:54.415234 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.415200 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2"] Jul 16 21:15:54.415634 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.415494 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb96891-a061-4969-9368-ad791fbaba28" containerName="util" Jul 16 21:15:54.415634 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.415504 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb96891-a061-4969-9368-ad791fbaba28" containerName="util" Jul 16 21:15:54.415634 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.415517 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb96891-a061-4969-9368-ad791fbaba28" containerName="extract" Jul 16 21:15:54.415634 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.415522 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb96891-a061-4969-9368-ad791fbaba28" containerName="extract" Jul 16 21:15:54.415634 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.415531 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb96891-a061-4969-9368-ad791fbaba28" containerName="pull" Jul 16 21:15:54.415634 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.415536 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb96891-a061-4969-9368-ad791fbaba28" containerName="pull" Jul 16 21:15:54.415634 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.415582 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bb96891-a061-4969-9368-ad791fbaba28" containerName="extract" Jul 16 21:15:54.455653 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.455625 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2"] Jul 16 21:15:54.455809 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.455747 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" Jul 16 21:15:54.458780 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.458748 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-thrxk\"" Jul 16 21:15:54.459152 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.459121 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 21:15:54.459756 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.459743 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 21:15:54.524781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.524742 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/774602c5-cca7-405e-a0eb-2fbcb6f95478-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2\" (UID: \"774602c5-cca7-405e-a0eb-2fbcb6f95478\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" Jul 16 21:15:54.524958 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.524813 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/774602c5-cca7-405e-a0eb-2fbcb6f95478-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2\" (UID: \"774602c5-cca7-405e-a0eb-2fbcb6f95478\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" Jul 16 21:15:54.524958 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.524845 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsqvk\" (UniqueName: \"kubernetes.io/projected/774602c5-cca7-405e-a0eb-2fbcb6f95478-kube-api-access-bsqvk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2\" (UID: \"774602c5-cca7-405e-a0eb-2fbcb6f95478\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" Jul 16 21:15:54.625266 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.625234 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/774602c5-cca7-405e-a0eb-2fbcb6f95478-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2\" (UID: \"774602c5-cca7-405e-a0eb-2fbcb6f95478\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" Jul 16 21:15:54.625266 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.625268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsqvk\" (UniqueName: \"kubernetes.io/projected/774602c5-cca7-405e-a0eb-2fbcb6f95478-kube-api-access-bsqvk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2\" (UID: \"774602c5-cca7-405e-a0eb-2fbcb6f95478\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" Jul 16 21:15:54.625461 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.625308 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/774602c5-cca7-405e-a0eb-2fbcb6f95478-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2\" (UID: \"774602c5-cca7-405e-a0eb-2fbcb6f95478\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" Jul 16 21:15:54.625611 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.625591 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/774602c5-cca7-405e-a0eb-2fbcb6f95478-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2\" (UID: \"774602c5-cca7-405e-a0eb-2fbcb6f95478\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" Jul 16 21:15:54.625654 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.625619 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/774602c5-cca7-405e-a0eb-2fbcb6f95478-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2\" (UID: \"774602c5-cca7-405e-a0eb-2fbcb6f95478\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" Jul 16 21:15:54.636038 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.636009 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsqvk\" (UniqueName: \"kubernetes.io/projected/774602c5-cca7-405e-a0eb-2fbcb6f95478-kube-api-access-bsqvk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2\" (UID: \"774602c5-cca7-405e-a0eb-2fbcb6f95478\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" Jul 16 21:15:54.766552 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.766471 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" Jul 16 21:15:54.946950 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.946919 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2"] Jul 16 21:15:54.949954 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:15:54.949929 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774602c5_cca7_405e_a0eb_2fbcb6f95478.slice/crio-1b9c34996ca81661c5036155700a5c78cf12fe6363e1e3dfae09163d38f8e929 WatchSource:0}: Error finding container 1b9c34996ca81661c5036155700a5c78cf12fe6363e1e3dfae09163d38f8e929: Status 404 returned error can't find the container with id 1b9c34996ca81661c5036155700a5c78cf12fe6363e1e3dfae09163d38f8e929 Jul 16 21:15:54.980779 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:54.980748 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" event={"ID":"774602c5-cca7-405e-a0eb-2fbcb6f95478","Type":"ContainerStarted","Data":"1b9c34996ca81661c5036155700a5c78cf12fe6363e1e3dfae09163d38f8e929"} Jul 16 21:15:55.986674 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:55.986592 2577 generic.go:356] "Generic (PLEG): container finished" podID="774602c5-cca7-405e-a0eb-2fbcb6f95478" containerID="4ac73f8329f56855dfb568af06d8f09c1f012b111e37b9d17b3164b1aa1686d5" exitCode=0 Jul 16 21:15:55.986674 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:55.986655 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" event={"ID":"774602c5-cca7-405e-a0eb-2fbcb6f95478","Type":"ContainerDied","Data":"4ac73f8329f56855dfb568af06d8f09c1f012b111e37b9d17b3164b1aa1686d5"} Jul 16 21:15:57.996107 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:57.996072 2577 generic.go:356] "Generic (PLEG): container finished" podID="774602c5-cca7-405e-a0eb-2fbcb6f95478" containerID="0df4fab2ee481ad07bcec5502ce22ea858aeaf413ed460c0a47e390a35e5c1b6" exitCode=0 Jul 16 21:15:57.996527 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:57.996132 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" event={"ID":"774602c5-cca7-405e-a0eb-2fbcb6f95478","Type":"ContainerDied","Data":"0df4fab2ee481ad07bcec5502ce22ea858aeaf413ed460c0a47e390a35e5c1b6"} Jul 16 21:15:59.003401 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:59.003365 2577 generic.go:356] "Generic (PLEG): container finished" podID="774602c5-cca7-405e-a0eb-2fbcb6f95478" containerID="de303c3cf60898b2351788b17bb71376459f4d84593051ea08ff30ac69c10393" exitCode=0 Jul 16 21:15:59.003767 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:15:59.003449 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" event={"ID":"774602c5-cca7-405e-a0eb-2fbcb6f95478","Type":"ContainerDied","Data":"de303c3cf60898b2351788b17bb71376459f4d84593051ea08ff30ac69c10393"} Jul 16 21:16:00.139200 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:00.139167 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" Jul 16 21:16:00.174596 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:00.174567 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsqvk\" (UniqueName: \"kubernetes.io/projected/774602c5-cca7-405e-a0eb-2fbcb6f95478-kube-api-access-bsqvk\") pod \"774602c5-cca7-405e-a0eb-2fbcb6f95478\" (UID: \"774602c5-cca7-405e-a0eb-2fbcb6f95478\") " Jul 16 21:16:00.174750 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:00.174658 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/774602c5-cca7-405e-a0eb-2fbcb6f95478-util\") pod \"774602c5-cca7-405e-a0eb-2fbcb6f95478\" (UID: \"774602c5-cca7-405e-a0eb-2fbcb6f95478\") " Jul 16 21:16:00.174750 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:00.174677 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/774602c5-cca7-405e-a0eb-2fbcb6f95478-bundle\") pod \"774602c5-cca7-405e-a0eb-2fbcb6f95478\" (UID: \"774602c5-cca7-405e-a0eb-2fbcb6f95478\") " Jul 16 21:16:00.175433 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:00.175399 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774602c5-cca7-405e-a0eb-2fbcb6f95478-bundle" (OuterVolumeSpecName: "bundle") pod "774602c5-cca7-405e-a0eb-2fbcb6f95478" (UID: "774602c5-cca7-405e-a0eb-2fbcb6f95478"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:16:00.176587 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:00.176559 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774602c5-cca7-405e-a0eb-2fbcb6f95478-kube-api-access-bsqvk" (OuterVolumeSpecName: "kube-api-access-bsqvk") pod "774602c5-cca7-405e-a0eb-2fbcb6f95478" (UID: "774602c5-cca7-405e-a0eb-2fbcb6f95478"). InnerVolumeSpecName "kube-api-access-bsqvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:16:00.199924 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:00.199882 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774602c5-cca7-405e-a0eb-2fbcb6f95478-util" (OuterVolumeSpecName: "util") pod "774602c5-cca7-405e-a0eb-2fbcb6f95478" (UID: "774602c5-cca7-405e-a0eb-2fbcb6f95478"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:16:00.275346 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:00.275268 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/774602c5-cca7-405e-a0eb-2fbcb6f95478-util\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:16:00.275346 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:00.275292 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/774602c5-cca7-405e-a0eb-2fbcb6f95478-bundle\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:16:00.275346 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:00.275303 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsqvk\" (UniqueName: \"kubernetes.io/projected/774602c5-cca7-405e-a0eb-2fbcb6f95478-kube-api-access-bsqvk\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:16:01.018758 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:01.018732 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" Jul 16 21:16:01.018949 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:01.018759 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af758v2" event={"ID":"774602c5-cca7-405e-a0eb-2fbcb6f95478","Type":"ContainerDied","Data":"1b9c34996ca81661c5036155700a5c78cf12fe6363e1e3dfae09163d38f8e929"} Jul 16 21:16:01.018949 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:01.018803 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b9c34996ca81661c5036155700a5c78cf12fe6363e1e3dfae09163d38f8e929" Jul 16 21:16:15.075206 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.074527 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn"] Jul 16 21:16:15.075820 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.075799 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="774602c5-cca7-405e-a0eb-2fbcb6f95478" containerName="util" Jul 16 21:16:15.076211 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.075905 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="774602c5-cca7-405e-a0eb-2fbcb6f95478" containerName="util" Jul 16 21:16:15.076211 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.075934 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="774602c5-cca7-405e-a0eb-2fbcb6f95478" containerName="pull" Jul 16 21:16:15.076211 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.075943 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="774602c5-cca7-405e-a0eb-2fbcb6f95478" containerName="pull" Jul 16 21:16:15.076211 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.075977 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="774602c5-cca7-405e-a0eb-2fbcb6f95478" containerName="extract" Jul 16 21:16:15.076211 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.075994 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="774602c5-cca7-405e-a0eb-2fbcb6f95478" containerName="extract" Jul 16 21:16:15.076211 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.076168 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="774602c5-cca7-405e-a0eb-2fbcb6f95478" containerName="extract" Jul 16 21:16:15.080039 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.080012 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.085831 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.085807 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 21:16:15.085948 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.085839 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 21:16:15.086496 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.086257 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 21:16:15.086954 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.086935 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-drfg4\"" Jul 16 21:16:15.087174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.087157 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 21:16:15.163097 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.163065 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn"] Jul 16 21:16:15.201503 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.201469 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3-webhook-cert\") pod \"opendatahub-operator-controller-manager-9f7d7dc89-6jjxn\" (UID: \"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3\") " pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.201675 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.201511 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8475\" (UniqueName: \"kubernetes.io/projected/f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3-kube-api-access-q8475\") pod \"opendatahub-operator-controller-manager-9f7d7dc89-6jjxn\" (UID: \"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3\") " pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.201675 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.201590 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9f7d7dc89-6jjxn\" (UID: \"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3\") " pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.201675 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.201620 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3-tmp\") pod \"opendatahub-operator-controller-manager-9f7d7dc89-6jjxn\" (UID: \"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3\") " pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.201675 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.201656 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3-manifests\") pod \"opendatahub-operator-controller-manager-9f7d7dc89-6jjxn\" (UID: \"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3\") " pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.302351 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.302316 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3-webhook-cert\") pod \"opendatahub-operator-controller-manager-9f7d7dc89-6jjxn\" (UID: \"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3\") " pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.302538 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.302363 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8475\" (UniqueName: \"kubernetes.io/projected/f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3-kube-api-access-q8475\") pod \"opendatahub-operator-controller-manager-9f7d7dc89-6jjxn\" (UID: \"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3\") " pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.302538 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.302420 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9f7d7dc89-6jjxn\" (UID: \"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3\") " pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.302538 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.302453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3-tmp\") pod \"opendatahub-operator-controller-manager-9f7d7dc89-6jjxn\" (UID: \"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3\") " pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.302538 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.302490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3-manifests\") pod \"opendatahub-operator-controller-manager-9f7d7dc89-6jjxn\" (UID: \"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3\") " pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.302828 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.302792 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3-manifests\") pod \"opendatahub-operator-controller-manager-9f7d7dc89-6jjxn\" (UID: \"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3\") " pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.304689 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.304667 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3-tmp\") pod \"opendatahub-operator-controller-manager-9f7d7dc89-6jjxn\" (UID: \"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3\") " pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.304949 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.304933 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9f7d7dc89-6jjxn\" (UID: \"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3\") " pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.304995 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.304978 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3-webhook-cert\") pod \"opendatahub-operator-controller-manager-9f7d7dc89-6jjxn\" (UID: \"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3\") " pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.330072 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.330011 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8475\" (UniqueName: \"kubernetes.io/projected/f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3-kube-api-access-q8475\") pod \"opendatahub-operator-controller-manager-9f7d7dc89-6jjxn\" (UID: \"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3\") " pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.389687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.389655 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:15.554572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:15.554543 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn"] Jul 16 21:16:15.557313 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:16:15.557283 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b2bde4_3a9c_4891_9d9b_dbe5182e5eb3.slice/crio-02d4f683179747285e2945f8ca9b2a061c54972e62aa14a6688cfa3fa9225959 WatchSource:0}: Error finding container 02d4f683179747285e2945f8ca9b2a061c54972e62aa14a6688cfa3fa9225959: Status 404 returned error can't find the container with id 02d4f683179747285e2945f8ca9b2a061c54972e62aa14a6688cfa3fa9225959 Jul 16 21:16:16.087960 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.087921 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" event={"ID":"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3","Type":"ContainerStarted","Data":"02d4f683179747285e2945f8ca9b2a061c54972e62aa14a6688cfa3fa9225959"} Jul 16 21:16:16.644771 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.644738 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl"] Jul 16 21:16:16.647085 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.647066 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" Jul 16 21:16:16.652497 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.652321 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 16 21:16:16.652742 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.652714 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:16:16.652843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.652718 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 16 21:16:16.653404 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.653370 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 16 21:16:16.653506 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.653473 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rhnrc\"" Jul 16 21:16:16.661529 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.661458 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 21:16:16.713478 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.713445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8fb8205a-b1e8-424f-b3f0-a56664002770-metrics-cert\") pod \"lws-controller-manager-68bcfc7f9d-kvkhl\" (UID: \"8fb8205a-b1e8-424f-b3f0-a56664002770\") " pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" Jul 16 21:16:16.713651 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.713491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fb8205a-b1e8-424f-b3f0-a56664002770-cert\") pod \"lws-controller-manager-68bcfc7f9d-kvkhl\" (UID: \"8fb8205a-b1e8-424f-b3f0-a56664002770\") " pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" Jul 16 21:16:16.713651 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.713635 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8fb8205a-b1e8-424f-b3f0-a56664002770-manager-config\") pod \"lws-controller-manager-68bcfc7f9d-kvkhl\" (UID: \"8fb8205a-b1e8-424f-b3f0-a56664002770\") " pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" Jul 16 21:16:16.713774 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.713689 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6z9p\" (UniqueName: \"kubernetes.io/projected/8fb8205a-b1e8-424f-b3f0-a56664002770-kube-api-access-d6z9p\") pod \"lws-controller-manager-68bcfc7f9d-kvkhl\" (UID: \"8fb8205a-b1e8-424f-b3f0-a56664002770\") " pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" Jul 16 21:16:16.722898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.722858 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl"] Jul 16 21:16:16.815202 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.815157 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8fb8205a-b1e8-424f-b3f0-a56664002770-manager-config\") pod \"lws-controller-manager-68bcfc7f9d-kvkhl\" (UID: \"8fb8205a-b1e8-424f-b3f0-a56664002770\") " pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" Jul 16 21:16:16.815354 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.815232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6z9p\" (UniqueName: \"kubernetes.io/projected/8fb8205a-b1e8-424f-b3f0-a56664002770-kube-api-access-d6z9p\") pod \"lws-controller-manager-68bcfc7f9d-kvkhl\" (UID: \"8fb8205a-b1e8-424f-b3f0-a56664002770\") " pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" Jul 16 21:16:16.815354 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.815337 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8fb8205a-b1e8-424f-b3f0-a56664002770-metrics-cert\") pod \"lws-controller-manager-68bcfc7f9d-kvkhl\" (UID: \"8fb8205a-b1e8-424f-b3f0-a56664002770\") " pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" Jul 16 21:16:16.815485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.815370 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fb8205a-b1e8-424f-b3f0-a56664002770-cert\") pod \"lws-controller-manager-68bcfc7f9d-kvkhl\" (UID: \"8fb8205a-b1e8-424f-b3f0-a56664002770\") " pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" Jul 16 21:16:16.816656 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.816600 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8fb8205a-b1e8-424f-b3f0-a56664002770-manager-config\") pod \"lws-controller-manager-68bcfc7f9d-kvkhl\" (UID: \"8fb8205a-b1e8-424f-b3f0-a56664002770\") " pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" Jul 16 21:16:16.819535 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.819462 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fb8205a-b1e8-424f-b3f0-a56664002770-cert\") pod \"lws-controller-manager-68bcfc7f9d-kvkhl\" (UID: \"8fb8205a-b1e8-424f-b3f0-a56664002770\") " pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" Jul 16 21:16:16.819535 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.819492 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8fb8205a-b1e8-424f-b3f0-a56664002770-metrics-cert\") pod \"lws-controller-manager-68bcfc7f9d-kvkhl\" (UID: \"8fb8205a-b1e8-424f-b3f0-a56664002770\") " pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" Jul 16 21:16:16.833797 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.833750 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6z9p\" (UniqueName: \"kubernetes.io/projected/8fb8205a-b1e8-424f-b3f0-a56664002770-kube-api-access-d6z9p\") pod \"lws-controller-manager-68bcfc7f9d-kvkhl\" (UID: \"8fb8205a-b1e8-424f-b3f0-a56664002770\") " pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" Jul 16 21:16:16.958877 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:16.958790 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" Jul 16 21:16:17.165131 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:17.165102 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl"] Jul 16 21:16:18.669148 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:16:18.669107 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb8205a_b1e8_424f_b3f0_a56664002770.slice/crio-c8104eca601ab75859f5062c430b96fa767c42bb657b228f8b58e25186dad2b5 WatchSource:0}: Error finding container c8104eca601ab75859f5062c430b96fa767c42bb657b228f8b58e25186dad2b5: Status 404 returned error can't find the container with id c8104eca601ab75859f5062c430b96fa767c42bb657b228f8b58e25186dad2b5 Jul 16 21:16:19.104056 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:19.103975 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" event={"ID":"8fb8205a-b1e8-424f-b3f0-a56664002770","Type":"ContainerStarted","Data":"c8104eca601ab75859f5062c430b96fa767c42bb657b228f8b58e25186dad2b5"} Jul 16 21:16:19.105461 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:19.105437 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" event={"ID":"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3","Type":"ContainerStarted","Data":"6411ef942e0de17dfb7bdde01439a968bd41157684ec6a98d9fce51435aec5be"} Jul 16 21:16:21.116643 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:21.116612 2577 generic.go:356] "Generic (PLEG): container finished" podID="f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3" containerID="6411ef942e0de17dfb7bdde01439a968bd41157684ec6a98d9fce51435aec5be" exitCode=0 Jul 16 21:16:21.117159 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:21.116685 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" event={"ID":"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3","Type":"ContainerDied","Data":"6411ef942e0de17dfb7bdde01439a968bd41157684ec6a98d9fce51435aec5be"} Jul 16 21:16:21.118287 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:21.118265 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" event={"ID":"8fb8205a-b1e8-424f-b3f0-a56664002770","Type":"ContainerStarted","Data":"69c8ec99e0dd72b735b0ee4a2f4006e36fc7cb905a1a9e3deb33b0317fd8fa19"} Jul 16 21:16:21.118421 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:21.118402 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" Jul 16 21:16:21.214102 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:21.214051 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" podStartSLOduration=3.7377409310000003 podStartE2EDuration="5.214034801s" totalImagesPullingTime="1.47629387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:16:21.213155559 +0000 UTC m=+397.388263750" watchObservedRunningTime="2026-07-16 21:16:21.214034801 +0000 UTC m=+397.389142992" Jul 16 21:16:22.124507 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:22.124428 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" event={"ID":"f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3","Type":"ContainerStarted","Data":"46070615acc5520ea584fa36ef5de30ca4caeafacad3b9fe0062e143ca5c5cb5"} Jul 16 21:16:22.124892 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:22.124698 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:22.167937 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:22.167887 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" podStartSLOduration=2.180352639 podStartE2EDuration="7.167871007s" totalImagesPullingTime="3.871590217s" totalInitContainerRuntime="1.115928151s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:16:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:16:22.165712884 +0000 UTC m=+398.340821075" watchObservedRunningTime="2026-07-16 21:16:22.167871007 +0000 UTC m=+398.342979198" Jul 16 21:16:32.126875 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:32.126839 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68bcfc7f9d-kvkhl" Jul 16 21:16:33.131005 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:33.130976 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9f7d7dc89-6jjxn" Jul 16 21:16:50.014398 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.014363 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf"] Jul 16 21:16:50.017708 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.017692 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.022254 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.022232 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 21:16:50.022254 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.022232 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-rdz5p\"" Jul 16 21:16:50.022417 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.022339 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 21:16:50.023357 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.023341 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 21:16:50.058774 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.058744 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf"] Jul 16 21:16:50.107020 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.106985 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb922b04-96b8-401f-8610-017771c0b3b8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.107210 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.107033 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9wlm\" (UniqueName: \"kubernetes.io/projected/eb922b04-96b8-401f-8610-017771c0b3b8-kube-api-access-b9wlm\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.107210 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.107091 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb922b04-96b8-401f-8610-017771c0b3b8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.107210 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.107116 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb922b04-96b8-401f-8610-017771c0b3b8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.107210 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.107175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb922b04-96b8-401f-8610-017771c0b3b8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.107370 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.107267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb922b04-96b8-401f-8610-017771c0b3b8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.107370 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.107308 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb922b04-96b8-401f-8610-017771c0b3b8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.107370 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.107335 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb922b04-96b8-401f-8610-017771c0b3b8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.107370 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.107366 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb922b04-96b8-401f-8610-017771c0b3b8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.208290 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.208259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb922b04-96b8-401f-8610-017771c0b3b8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.208480 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.208298 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9wlm\" (UniqueName: \"kubernetes.io/projected/eb922b04-96b8-401f-8610-017771c0b3b8-kube-api-access-b9wlm\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.208480 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.208327 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb922b04-96b8-401f-8610-017771c0b3b8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.208480 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.208437 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb922b04-96b8-401f-8610-017771c0b3b8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.208665 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.208485 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb922b04-96b8-401f-8610-017771c0b3b8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.208665 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.208529 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb922b04-96b8-401f-8610-017771c0b3b8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.208665 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.208577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb922b04-96b8-401f-8610-017771c0b3b8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.208665 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.208601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb922b04-96b8-401f-8610-017771c0b3b8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.208665 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.208617 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb922b04-96b8-401f-8610-017771c0b3b8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.208665 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.208655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb922b04-96b8-401f-8610-017771c0b3b8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.208939 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.208771 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb922b04-96b8-401f-8610-017771c0b3b8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.208939 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.208826 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb922b04-96b8-401f-8610-017771c0b3b8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.209003 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.208965 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb922b04-96b8-401f-8610-017771c0b3b8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.209168 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.209153 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb922b04-96b8-401f-8610-017771c0b3b8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.210754 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.210704 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb922b04-96b8-401f-8610-017771c0b3b8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.211379 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.211356 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb922b04-96b8-401f-8610-017771c0b3b8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.239579 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.239555 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb922b04-96b8-401f-8610-017771c0b3b8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.242990 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.242966 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9wlm\" (UniqueName: \"kubernetes.io/projected/eb922b04-96b8-401f-8610-017771c0b3b8-kube-api-access-b9wlm\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pfstf\" (UID: \"eb922b04-96b8-401f-8610-017771c0b3b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.327390 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.327313 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:50.543834 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:16:50.543801 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb922b04_96b8_401f_8610_017771c0b3b8.slice/crio-594799ab039a40b303aa1b0b21e19ab34eeb0994943924f037c0b9da02bb5bb9 WatchSource:0}: Error finding container 594799ab039a40b303aa1b0b21e19ab34eeb0994943924f037c0b9da02bb5bb9: Status 404 returned error can't find the container with id 594799ab039a40b303aa1b0b21e19ab34eeb0994943924f037c0b9da02bb5bb9 Jul 16 21:16:50.554130 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.554108 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf"] Jul 16 21:16:50.805950 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.805912 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl"] Jul 16 21:16:50.809257 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.809234 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:50.814772 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.814749 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 16 21:16:50.815289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.815266 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2x9gh\"" Jul 16 21:16:50.819052 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.819032 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 16 21:16:50.828199 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.828155 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 16 21:16:50.843802 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.843778 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl"] Jul 16 21:16:50.914934 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.914891 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvjmk\" (UniqueName: \"kubernetes.io/projected/42846e1e-6fd3-4bc3-9ec2-c558feb88415-kube-api-access-fvjmk\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:50.915116 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.914998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/42846e1e-6fd3-4bc3-9ec2-c558feb88415-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:50.915116 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.915036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/42846e1e-6fd3-4bc3-9ec2-c558feb88415-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:50.915116 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.915064 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42846e1e-6fd3-4bc3-9ec2-c558feb88415-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:50.915116 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.915085 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/42846e1e-6fd3-4bc3-9ec2-c558feb88415-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:50.915116 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.915103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42846e1e-6fd3-4bc3-9ec2-c558feb88415-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:50.915360 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:50.915277 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/42846e1e-6fd3-4bc3-9ec2-c558feb88415-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:51.016133 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.016102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/42846e1e-6fd3-4bc3-9ec2-c558feb88415-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:51.016612 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.016144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvjmk\" (UniqueName: \"kubernetes.io/projected/42846e1e-6fd3-4bc3-9ec2-c558feb88415-kube-api-access-fvjmk\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:51.016612 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.016240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/42846e1e-6fd3-4bc3-9ec2-c558feb88415-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:51.016612 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.016289 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/42846e1e-6fd3-4bc3-9ec2-c558feb88415-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:51.016612 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.016320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42846e1e-6fd3-4bc3-9ec2-c558feb88415-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:51.016612 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.016433 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/42846e1e-6fd3-4bc3-9ec2-c558feb88415-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:51.016612 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.016478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42846e1e-6fd3-4bc3-9ec2-c558feb88415-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:51.017158 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.017133 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/42846e1e-6fd3-4bc3-9ec2-c558feb88415-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:51.018462 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.018438 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/42846e1e-6fd3-4bc3-9ec2-c558feb88415-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:51.018594 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.018576 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/42846e1e-6fd3-4bc3-9ec2-c558feb88415-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:51.018875 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.018846 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/42846e1e-6fd3-4bc3-9ec2-c558feb88415-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:51.018950 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.018919 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42846e1e-6fd3-4bc3-9ec2-c558feb88415-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:51.033006 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.032983 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42846e1e-6fd3-4bc3-9ec2-c558feb88415-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:51.033109 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.033015 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvjmk\" (UniqueName: \"kubernetes.io/projected/42846e1e-6fd3-4bc3-9ec2-c558feb88415-kube-api-access-fvjmk\") pod \"istiod-openshift-gateway-94bb8fbfd-585pl\" (UID: \"42846e1e-6fd3-4bc3-9ec2-c558feb88415\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:51.118476 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.118393 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:51.253680 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.253646 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" event={"ID":"eb922b04-96b8-401f-8610-017771c0b3b8","Type":"ContainerStarted","Data":"594799ab039a40b303aa1b0b21e19ab34eeb0994943924f037c0b9da02bb5bb9"} Jul 16 21:16:51.332067 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:51.332040 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl"] Jul 16 21:16:51.361591 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:16:51.361547 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42846e1e_6fd3_4bc3_9ec2_c558feb88415.slice/crio-91922bc5663ea614e942ef3d7f78edabfd0dd244f33fd2e9966c7486de6ca5ce WatchSource:0}: Error finding container 91922bc5663ea614e942ef3d7f78edabfd0dd244f33fd2e9966c7486de6ca5ce: Status 404 returned error can't find the container with id 91922bc5663ea614e942ef3d7f78edabfd0dd244f33fd2e9966c7486de6ca5ce Jul 16 21:16:52.260593 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:52.260554 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" event={"ID":"42846e1e-6fd3-4bc3-9ec2-c558feb88415","Type":"ContainerStarted","Data":"91922bc5663ea614e942ef3d7f78edabfd0dd244f33fd2e9966c7486de6ca5ce"} Jul 16 21:16:54.808556 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:54.808511 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:16:54.808860 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:54.808592 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:16:54.812115 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:54.812077 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:16:54.812250 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:54.812154 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:16:54.812304 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:54.812233 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:16:55.280654 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:55.280616 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" event={"ID":"42846e1e-6fd3-4bc3-9ec2-c558feb88415","Type":"ContainerStarted","Data":"201a2ab350348694822926b24e0e5c580c2f8be7d3d3ec7ba8065425ce203293"} Jul 16 21:16:55.280876 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:55.280685 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:16:55.282419 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:55.282382 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" event={"ID":"eb922b04-96b8-401f-8610-017771c0b3b8","Type":"ContainerStarted","Data":"b536401bf2620d7452a2cbb711891a177bca8174c324cc151a6fcb015d364774"} Jul 16 21:16:55.282813 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:55.282792 2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-585pl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 21:16:55.282897 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:55.282842 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" podUID="42846e1e-6fd3-4bc3-9ec2-c558feb88415" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 21:16:55.328055 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:55.328021 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:55.329270 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:55.329249 2577 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6576859cf4pfstf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 16 21:16:55.329350 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:55.329296 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" podUID="eb922b04-96b8-401f-8610-017771c0b3b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 16 21:16:55.363357 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:55.363303 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" podStartSLOduration=1.918650392 podStartE2EDuration="5.363287286s" totalImagesPullingTime="3.444636894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:16:55.353153831 +0000 UTC m=+431.528262023" watchObservedRunningTime="2026-07-16 21:16:55.363287286 +0000 UTC m=+431.538395477" Jul 16 21:16:56.288492 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:56.288457 2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-585pl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 21:16:56.288838 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:56.288525 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" podUID="42846e1e-6fd3-4bc3-9ec2-c558feb88415" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 21:16:56.331860 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:56.331826 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:56.371875 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:56.371827 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" podStartSLOduration=3.105710395 podStartE2EDuration="7.371810946s" totalImagesPullingTime="4.266100551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:16:55.528349476 +0000 UTC m=+431.703457691" watchObservedRunningTime="2026-07-16 21:16:56.371810946 +0000 UTC m=+432.546919135" Jul 16 21:16:57.292129 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:57.292102 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:57.292933 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:57.292916 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pfstf" Jul 16 21:16:59.288803 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:16:59.288773 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-585pl" Jul 16 21:17:39.986105 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:39.986066 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-546ddb9b4f-zxhzc"] Jul 16 21:17:53.310788 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.310755 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs"] Jul 16 21:17:53.313508 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.313490 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" Jul 16 21:17:53.316500 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.316478 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 21:17:53.317656 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.317638 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nnslt\"" Jul 16 21:17:53.317656 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.317643 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 21:17:53.323828 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.323806 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs"] Jul 16 21:17:53.352479 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.352448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ee8cd36-2859-4bf3-a453-e468eb83048d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs\" (UID: \"8ee8cd36-2859-4bf3-a453-e468eb83048d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" Jul 16 21:17:53.352641 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.352493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ee8cd36-2859-4bf3-a453-e468eb83048d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs\" (UID: \"8ee8cd36-2859-4bf3-a453-e468eb83048d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" Jul 16 21:17:53.352641 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.352608 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpfbq\" (UniqueName: \"kubernetes.io/projected/8ee8cd36-2859-4bf3-a453-e468eb83048d-kube-api-access-qpfbq\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs\" (UID: \"8ee8cd36-2859-4bf3-a453-e468eb83048d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" Jul 16 21:17:53.453599 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.453563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ee8cd36-2859-4bf3-a453-e468eb83048d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs\" (UID: \"8ee8cd36-2859-4bf3-a453-e468eb83048d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" Jul 16 21:17:53.453770 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.453616 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ee8cd36-2859-4bf3-a453-e468eb83048d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs\" (UID: \"8ee8cd36-2859-4bf3-a453-e468eb83048d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" Jul 16 21:17:53.453770 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.453705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpfbq\" (UniqueName: \"kubernetes.io/projected/8ee8cd36-2859-4bf3-a453-e468eb83048d-kube-api-access-qpfbq\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs\" (UID: \"8ee8cd36-2859-4bf3-a453-e468eb83048d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" Jul 16 21:17:53.453984 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.453966 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ee8cd36-2859-4bf3-a453-e468eb83048d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs\" (UID: \"8ee8cd36-2859-4bf3-a453-e468eb83048d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" Jul 16 21:17:53.454048 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.454025 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ee8cd36-2859-4bf3-a453-e468eb83048d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs\" (UID: \"8ee8cd36-2859-4bf3-a453-e468eb83048d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" Jul 16 21:17:53.463411 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.463387 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpfbq\" (UniqueName: \"kubernetes.io/projected/8ee8cd36-2859-4bf3-a453-e468eb83048d-kube-api-access-qpfbq\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs\" (UID: \"8ee8cd36-2859-4bf3-a453-e468eb83048d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" Jul 16 21:17:53.623077 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.622974 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" Jul 16 21:17:53.705492 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.704994 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5"] Jul 16 21:17:53.729050 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.727943 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5"] Jul 16 21:17:53.729050 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.728065 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" Jul 16 21:17:53.791106 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.791082 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs"] Jul 16 21:17:53.792764 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:17:53.792730 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee8cd36_2859_4bf3_a453_e468eb83048d.slice/crio-6aed38fcf7a27a35333fc321f48b08db017400d4e22ce1537042b7f7f3ceb00a WatchSource:0}: Error finding container 6aed38fcf7a27a35333fc321f48b08db017400d4e22ce1537042b7f7f3ceb00a: Status 404 returned error can't find the container with id 6aed38fcf7a27a35333fc321f48b08db017400d4e22ce1537042b7f7f3ceb00a Jul 16 21:17:53.858449 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.858403 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ec078ff-b939-4e9a-a206-81ea686b278b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5\" (UID: \"6ec078ff-b939-4e9a-a206-81ea686b278b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" Jul 16 21:17:53.858578 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.858515 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h64tx\" (UniqueName: \"kubernetes.io/projected/6ec078ff-b939-4e9a-a206-81ea686b278b-kube-api-access-h64tx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5\" (UID: \"6ec078ff-b939-4e9a-a206-81ea686b278b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" Jul 16 21:17:53.858634 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.858573 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ec078ff-b939-4e9a-a206-81ea686b278b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5\" (UID: \"6ec078ff-b939-4e9a-a206-81ea686b278b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" Jul 16 21:17:53.959994 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.959963 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h64tx\" (UniqueName: \"kubernetes.io/projected/6ec078ff-b939-4e9a-a206-81ea686b278b-kube-api-access-h64tx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5\" (UID: \"6ec078ff-b939-4e9a-a206-81ea686b278b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" Jul 16 21:17:53.960123 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.960020 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ec078ff-b939-4e9a-a206-81ea686b278b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5\" (UID: \"6ec078ff-b939-4e9a-a206-81ea686b278b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" Jul 16 21:17:53.960123 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.960075 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ec078ff-b939-4e9a-a206-81ea686b278b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5\" (UID: \"6ec078ff-b939-4e9a-a206-81ea686b278b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" Jul 16 21:17:53.960413 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.960394 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ec078ff-b939-4e9a-a206-81ea686b278b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5\" (UID: \"6ec078ff-b939-4e9a-a206-81ea686b278b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" Jul 16 21:17:53.960456 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.960434 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ec078ff-b939-4e9a-a206-81ea686b278b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5\" (UID: \"6ec078ff-b939-4e9a-a206-81ea686b278b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" Jul 16 21:17:53.969976 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:53.969959 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h64tx\" (UniqueName: \"kubernetes.io/projected/6ec078ff-b939-4e9a-a206-81ea686b278b-kube-api-access-h64tx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5\" (UID: \"6ec078ff-b939-4e9a-a206-81ea686b278b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" Jul 16 21:17:54.040643 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.040614 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" Jul 16 21:17:54.115329 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.115296 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr"] Jul 16 21:17:54.118646 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.118625 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" Jul 16 21:17:54.128789 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.127801 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr"] Jul 16 21:17:54.162679 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.162606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92841e70-433a-42e1-8313-5419ab6f6399-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr\" (UID: \"92841e70-433a-42e1-8313-5419ab6f6399\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" Jul 16 21:17:54.162814 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.162678 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92841e70-433a-42e1-8313-5419ab6f6399-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr\" (UID: \"92841e70-433a-42e1-8313-5419ab6f6399\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" Jul 16 21:17:54.162814 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.162740 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7dfz\" (UniqueName: \"kubernetes.io/projected/92841e70-433a-42e1-8313-5419ab6f6399-kube-api-access-f7dfz\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr\" (UID: \"92841e70-433a-42e1-8313-5419ab6f6399\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" Jul 16 21:17:54.180111 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.180086 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5"] Jul 16 21:17:54.181952 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:17:54.181923 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec078ff_b939_4e9a_a206_81ea686b278b.slice/crio-242f7a4e4436542d67cbb143786a707b724dcd8a4f9eba6e8ae4ca629d150845 WatchSource:0}: Error finding container 242f7a4e4436542d67cbb143786a707b724dcd8a4f9eba6e8ae4ca629d150845: Status 404 returned error can't find the container with id 242f7a4e4436542d67cbb143786a707b724dcd8a4f9eba6e8ae4ca629d150845 Jul 16 21:17:54.263459 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.263426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7dfz\" (UniqueName: \"kubernetes.io/projected/92841e70-433a-42e1-8313-5419ab6f6399-kube-api-access-f7dfz\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr\" (UID: \"92841e70-433a-42e1-8313-5419ab6f6399\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" Jul 16 21:17:54.263612 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.263503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92841e70-433a-42e1-8313-5419ab6f6399-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr\" (UID: \"92841e70-433a-42e1-8313-5419ab6f6399\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" Jul 16 21:17:54.263612 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.263532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92841e70-433a-42e1-8313-5419ab6f6399-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr\" (UID: \"92841e70-433a-42e1-8313-5419ab6f6399\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" Jul 16 21:17:54.263860 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.263844 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92841e70-433a-42e1-8313-5419ab6f6399-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr\" (UID: \"92841e70-433a-42e1-8313-5419ab6f6399\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" Jul 16 21:17:54.263972 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.263949 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92841e70-433a-42e1-8313-5419ab6f6399-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr\" (UID: \"92841e70-433a-42e1-8313-5419ab6f6399\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" Jul 16 21:17:54.273613 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.273584 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7dfz\" (UniqueName: \"kubernetes.io/projected/92841e70-433a-42e1-8313-5419ab6f6399-kube-api-access-f7dfz\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr\" (UID: \"92841e70-433a-42e1-8313-5419ab6f6399\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" Jul 16 21:17:54.432093 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.432002 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" Jul 16 21:17:54.543952 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.543915 2577 generic.go:356] "Generic (PLEG): container finished" podID="8ee8cd36-2859-4bf3-a453-e468eb83048d" containerID="92b420978f5a5201a7f99d562f37f5112a479314a9ff73dacab690d63acb6fda" exitCode=0 Jul 16 21:17:54.544129 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.544002 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" event={"ID":"8ee8cd36-2859-4bf3-a453-e468eb83048d","Type":"ContainerDied","Data":"92b420978f5a5201a7f99d562f37f5112a479314a9ff73dacab690d63acb6fda"} Jul 16 21:17:54.544129 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.544058 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" event={"ID":"8ee8cd36-2859-4bf3-a453-e468eb83048d","Type":"ContainerStarted","Data":"6aed38fcf7a27a35333fc321f48b08db017400d4e22ce1537042b7f7f3ceb00a"} Jul 16 21:17:54.546012 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.545986 2577 generic.go:356] "Generic (PLEG): container finished" podID="6ec078ff-b939-4e9a-a206-81ea686b278b" containerID="9089c11f11d957891a5ae3ec04480f31079054e46cfb3db4798d40c70de66014" exitCode=0 Jul 16 21:17:54.546105 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.546024 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" event={"ID":"6ec078ff-b939-4e9a-a206-81ea686b278b","Type":"ContainerDied","Data":"9089c11f11d957891a5ae3ec04480f31079054e46cfb3db4798d40c70de66014"} Jul 16 21:17:54.546105 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.546068 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" event={"ID":"6ec078ff-b939-4e9a-a206-81ea686b278b","Type":"ContainerStarted","Data":"242f7a4e4436542d67cbb143786a707b724dcd8a4f9eba6e8ae4ca629d150845"} Jul 16 21:17:54.619918 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.619893 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr"] Jul 16 21:17:54.621582 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:17:54.621556 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92841e70_433a_42e1_8313_5419ab6f6399.slice/crio-ec5ca480c1a8e027e3373ea467833323cbc878c8c34f199c91f4abce9c975b04 WatchSource:0}: Error finding container ec5ca480c1a8e027e3373ea467833323cbc878c8c34f199c91f4abce9c975b04: Status 404 returned error can't find the container with id ec5ca480c1a8e027e3373ea467833323cbc878c8c34f199c91f4abce9c975b04 Jul 16 21:17:54.834524 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.834489 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw"] Jul 16 21:17:54.837091 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.837075 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" Jul 16 21:17:54.856339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.856309 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw"] Jul 16 21:17:54.967943 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.967853 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2cbded0-5f0e-4865-a7d5-cd2619506273-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw\" (UID: \"f2cbded0-5f0e-4865-a7d5-cd2619506273\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" Jul 16 21:17:54.967943 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.967896 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2cbded0-5f0e-4865-a7d5-cd2619506273-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw\" (UID: \"f2cbded0-5f0e-4865-a7d5-cd2619506273\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" Jul 16 21:17:54.967943 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:54.967921 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk6hd\" (UniqueName: \"kubernetes.io/projected/f2cbded0-5f0e-4865-a7d5-cd2619506273-kube-api-access-wk6hd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw\" (UID: \"f2cbded0-5f0e-4865-a7d5-cd2619506273\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" Jul 16 21:17:55.069240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.069201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2cbded0-5f0e-4865-a7d5-cd2619506273-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw\" (UID: \"f2cbded0-5f0e-4865-a7d5-cd2619506273\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" Jul 16 21:17:55.069240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.069244 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2cbded0-5f0e-4865-a7d5-cd2619506273-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw\" (UID: \"f2cbded0-5f0e-4865-a7d5-cd2619506273\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" Jul 16 21:17:55.069484 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.069273 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk6hd\" (UniqueName: \"kubernetes.io/projected/f2cbded0-5f0e-4865-a7d5-cd2619506273-kube-api-access-wk6hd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw\" (UID: \"f2cbded0-5f0e-4865-a7d5-cd2619506273\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" Jul 16 21:17:55.069568 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.069547 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2cbded0-5f0e-4865-a7d5-cd2619506273-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw\" (UID: \"f2cbded0-5f0e-4865-a7d5-cd2619506273\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" Jul 16 21:17:55.069633 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.069569 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2cbded0-5f0e-4865-a7d5-cd2619506273-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw\" (UID: \"f2cbded0-5f0e-4865-a7d5-cd2619506273\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" Jul 16 21:17:55.082347 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.082318 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk6hd\" (UniqueName: \"kubernetes.io/projected/f2cbded0-5f0e-4865-a7d5-cd2619506273-kube-api-access-wk6hd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw\" (UID: \"f2cbded0-5f0e-4865-a7d5-cd2619506273\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" Jul 16 21:17:55.145724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.145692 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" Jul 16 21:17:55.294688 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.294656 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw"] Jul 16 21:17:55.305905 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:17:55.305879 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cbded0_5f0e_4865_a7d5_cd2619506273.slice/crio-1cd6f3254610f394415bc7ef43cdebfa4d23698f8ddd0d985ddd95217e062be6 WatchSource:0}: Error finding container 1cd6f3254610f394415bc7ef43cdebfa4d23698f8ddd0d985ddd95217e062be6: Status 404 returned error can't find the container with id 1cd6f3254610f394415bc7ef43cdebfa4d23698f8ddd0d985ddd95217e062be6 Jul 16 21:17:55.552443 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.552412 2577 generic.go:356] "Generic (PLEG): container finished" podID="8ee8cd36-2859-4bf3-a453-e468eb83048d" containerID="dab6025c7865d16d356386c3fd394346f070034b2465d6dd5d96650f96a50deb" exitCode=0 Jul 16 21:17:55.552855 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.552494 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" event={"ID":"8ee8cd36-2859-4bf3-a453-e468eb83048d","Type":"ContainerDied","Data":"dab6025c7865d16d356386c3fd394346f070034b2465d6dd5d96650f96a50deb"} Jul 16 21:17:55.554107 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.554011 2577 generic.go:356] "Generic (PLEG): container finished" podID="92841e70-433a-42e1-8313-5419ab6f6399" containerID="ad0099a6140cce2d11899a1b58f787e8862672c6f83d2b79d922bdf68a0d787d" exitCode=0 Jul 16 21:17:55.554107 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.554059 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" event={"ID":"92841e70-433a-42e1-8313-5419ab6f6399","Type":"ContainerDied","Data":"ad0099a6140cce2d11899a1b58f787e8862672c6f83d2b79d922bdf68a0d787d"} Jul 16 21:17:55.554107 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.554106 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" event={"ID":"92841e70-433a-42e1-8313-5419ab6f6399","Type":"ContainerStarted","Data":"ec5ca480c1a8e027e3373ea467833323cbc878c8c34f199c91f4abce9c975b04"} Jul 16 21:17:55.556059 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.556037 2577 generic.go:356] "Generic (PLEG): container finished" podID="6ec078ff-b939-4e9a-a206-81ea686b278b" containerID="ecc80db0e79b054b10926103b1ea8594b3f6e77ef240f279a8ceddac0e923322" exitCode=0 Jul 16 21:17:55.556265 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.556113 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" event={"ID":"6ec078ff-b939-4e9a-a206-81ea686b278b","Type":"ContainerDied","Data":"ecc80db0e79b054b10926103b1ea8594b3f6e77ef240f279a8ceddac0e923322"} Jul 16 21:17:55.557816 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.557796 2577 generic.go:356] "Generic (PLEG): container finished" podID="f2cbded0-5f0e-4865-a7d5-cd2619506273" containerID="d26081a7bece3e893e4d7b82458b3bacc4fc77f0bc44b6e1a901201348de118a" exitCode=0 Jul 16 21:17:55.557916 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.557878 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" event={"ID":"f2cbded0-5f0e-4865-a7d5-cd2619506273","Type":"ContainerDied","Data":"d26081a7bece3e893e4d7b82458b3bacc4fc77f0bc44b6e1a901201348de118a"} Jul 16 21:17:55.557964 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:55.557921 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" event={"ID":"f2cbded0-5f0e-4865-a7d5-cd2619506273","Type":"ContainerStarted","Data":"1cd6f3254610f394415bc7ef43cdebfa4d23698f8ddd0d985ddd95217e062be6"} Jul 16 21:17:56.564874 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:56.564839 2577 generic.go:356] "Generic (PLEG): container finished" podID="8ee8cd36-2859-4bf3-a453-e468eb83048d" containerID="dc6136a39ce0279d2abf0073bb4df461927b0e329a130226081686c2e4e71a42" exitCode=0 Jul 16 21:17:56.565326 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:56.564920 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" event={"ID":"8ee8cd36-2859-4bf3-a453-e468eb83048d","Type":"ContainerDied","Data":"dc6136a39ce0279d2abf0073bb4df461927b0e329a130226081686c2e4e71a42"} Jul 16 21:17:56.566727 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:56.566705 2577 generic.go:356] "Generic (PLEG): container finished" podID="92841e70-433a-42e1-8313-5419ab6f6399" containerID="77d591a9e20ee5f02a6bb25de5751493b3dbeb2fa66e2ec708ab8f09a25b3f9c" exitCode=0 Jul 16 21:17:56.566836 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:56.566781 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" event={"ID":"92841e70-433a-42e1-8313-5419ab6f6399","Type":"ContainerDied","Data":"77d591a9e20ee5f02a6bb25de5751493b3dbeb2fa66e2ec708ab8f09a25b3f9c"} Jul 16 21:17:56.568849 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:56.568829 2577 generic.go:356] "Generic (PLEG): container finished" podID="6ec078ff-b939-4e9a-a206-81ea686b278b" containerID="70f55e8325dae41ec3e8f53a5a466868b550f156da53899f71319bad42dce367" exitCode=0 Jul 16 21:17:56.568966 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:56.568908 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" event={"ID":"6ec078ff-b939-4e9a-a206-81ea686b278b","Type":"ContainerDied","Data":"70f55e8325dae41ec3e8f53a5a466868b550f156da53899f71319bad42dce367"} Jul 16 21:17:56.570756 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:56.570736 2577 generic.go:356] "Generic (PLEG): container finished" podID="f2cbded0-5f0e-4865-a7d5-cd2619506273" containerID="d63f83063b0bd6b42f94bb0dcd68e6e51a6e9d0a29b2fde2ccfa94b4b02f10c3" exitCode=0 Jul 16 21:17:56.570828 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:56.570761 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" event={"ID":"f2cbded0-5f0e-4865-a7d5-cd2619506273","Type":"ContainerDied","Data":"d63f83063b0bd6b42f94bb0dcd68e6e51a6e9d0a29b2fde2ccfa94b4b02f10c3"} Jul 16 21:17:57.577668 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.577632 2577 generic.go:356] "Generic (PLEG): container finished" podID="f2cbded0-5f0e-4865-a7d5-cd2619506273" containerID="ccc8bf83594c9104e52d40b12c74dad5b381308beb4ad48fbfc671c7857f75ed" exitCode=0 Jul 16 21:17:57.578068 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.577719 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" event={"ID":"f2cbded0-5f0e-4865-a7d5-cd2619506273","Type":"ContainerDied","Data":"ccc8bf83594c9104e52d40b12c74dad5b381308beb4ad48fbfc671c7857f75ed"} Jul 16 21:17:57.580415 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.580392 2577 generic.go:356] "Generic (PLEG): container finished" podID="92841e70-433a-42e1-8313-5419ab6f6399" containerID="c8e2a3d91b9e3671f0d38c59d53430aead2532ea9eafe3659756890e2ddb637f" exitCode=0 Jul 16 21:17:57.580541 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.580454 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" event={"ID":"92841e70-433a-42e1-8313-5419ab6f6399","Type":"ContainerDied","Data":"c8e2a3d91b9e3671f0d38c59d53430aead2532ea9eafe3659756890e2ddb637f"} Jul 16 21:17:57.712625 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.712600 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" Jul 16 21:17:57.741342 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.741316 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" Jul 16 21:17:57.789793 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.789766 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ee8cd36-2859-4bf3-a453-e468eb83048d-util\") pod \"8ee8cd36-2859-4bf3-a453-e468eb83048d\" (UID: \"8ee8cd36-2859-4bf3-a453-e468eb83048d\") " Jul 16 21:17:57.789954 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.789832 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ec078ff-b939-4e9a-a206-81ea686b278b-bundle\") pod \"6ec078ff-b939-4e9a-a206-81ea686b278b\" (UID: \"6ec078ff-b939-4e9a-a206-81ea686b278b\") " Jul 16 21:17:57.789954 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.789889 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h64tx\" (UniqueName: \"kubernetes.io/projected/6ec078ff-b939-4e9a-a206-81ea686b278b-kube-api-access-h64tx\") pod \"6ec078ff-b939-4e9a-a206-81ea686b278b\" (UID: \"6ec078ff-b939-4e9a-a206-81ea686b278b\") " Jul 16 21:17:57.789954 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.789946 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ee8cd36-2859-4bf3-a453-e468eb83048d-bundle\") pod \"8ee8cd36-2859-4bf3-a453-e468eb83048d\" (UID: \"8ee8cd36-2859-4bf3-a453-e468eb83048d\") " Jul 16 21:17:57.790053 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.789999 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpfbq\" (UniqueName: \"kubernetes.io/projected/8ee8cd36-2859-4bf3-a453-e468eb83048d-kube-api-access-qpfbq\") pod \"8ee8cd36-2859-4bf3-a453-e468eb83048d\" (UID: \"8ee8cd36-2859-4bf3-a453-e468eb83048d\") " Jul 16 21:17:57.790053 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.790045 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ec078ff-b939-4e9a-a206-81ea686b278b-util\") pod \"6ec078ff-b939-4e9a-a206-81ea686b278b\" (UID: \"6ec078ff-b939-4e9a-a206-81ea686b278b\") " Jul 16 21:17:57.790608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.790562 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee8cd36-2859-4bf3-a453-e468eb83048d-bundle" (OuterVolumeSpecName: "bundle") pod "8ee8cd36-2859-4bf3-a453-e468eb83048d" (UID: "8ee8cd36-2859-4bf3-a453-e468eb83048d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:17:57.790745 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.790631 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec078ff-b939-4e9a-a206-81ea686b278b-bundle" (OuterVolumeSpecName: "bundle") pod "6ec078ff-b939-4e9a-a206-81ea686b278b" (UID: "6ec078ff-b939-4e9a-a206-81ea686b278b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:17:57.792398 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.792374 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee8cd36-2859-4bf3-a453-e468eb83048d-kube-api-access-qpfbq" (OuterVolumeSpecName: "kube-api-access-qpfbq") pod "8ee8cd36-2859-4bf3-a453-e468eb83048d" (UID: "8ee8cd36-2859-4bf3-a453-e468eb83048d"). InnerVolumeSpecName "kube-api-access-qpfbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:17:57.792488 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.792410 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec078ff-b939-4e9a-a206-81ea686b278b-kube-api-access-h64tx" (OuterVolumeSpecName: "kube-api-access-h64tx") pod "6ec078ff-b939-4e9a-a206-81ea686b278b" (UID: "6ec078ff-b939-4e9a-a206-81ea686b278b"). InnerVolumeSpecName "kube-api-access-h64tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:17:57.795196 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.795160 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee8cd36-2859-4bf3-a453-e468eb83048d-util" (OuterVolumeSpecName: "util") pod "8ee8cd36-2859-4bf3-a453-e468eb83048d" (UID: "8ee8cd36-2859-4bf3-a453-e468eb83048d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:17:57.795642 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.795624 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec078ff-b939-4e9a-a206-81ea686b278b-util" (OuterVolumeSpecName: "util") pod "6ec078ff-b939-4e9a-a206-81ea686b278b" (UID: "6ec078ff-b939-4e9a-a206-81ea686b278b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:17:57.891529 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.891494 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpfbq\" (UniqueName: \"kubernetes.io/projected/8ee8cd36-2859-4bf3-a453-e468eb83048d-kube-api-access-qpfbq\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:17:57.891529 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.891521 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ec078ff-b939-4e9a-a206-81ea686b278b-util\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:17:57.891529 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.891531 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ee8cd36-2859-4bf3-a453-e468eb83048d-util\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:17:57.891529 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.891539 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ec078ff-b939-4e9a-a206-81ea686b278b-bundle\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:17:57.891762 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.891547 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h64tx\" (UniqueName: \"kubernetes.io/projected/6ec078ff-b939-4e9a-a206-81ea686b278b-kube-api-access-h64tx\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:17:57.891762 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:57.891556 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ee8cd36-2859-4bf3-a453-e468eb83048d-bundle\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:17:58.587490 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.587459 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" Jul 16 21:17:58.587490 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.587481 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5" event={"ID":"6ec078ff-b939-4e9a-a206-81ea686b278b","Type":"ContainerDied","Data":"242f7a4e4436542d67cbb143786a707b724dcd8a4f9eba6e8ae4ca629d150845"} Jul 16 21:17:58.587991 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.587531 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="242f7a4e4436542d67cbb143786a707b724dcd8a4f9eba6e8ae4ca629d150845" Jul 16 21:17:58.589539 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.589513 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" Jul 16 21:17:58.589680 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.589564 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs" event={"ID":"8ee8cd36-2859-4bf3-a453-e468eb83048d","Type":"ContainerDied","Data":"6aed38fcf7a27a35333fc321f48b08db017400d4e22ce1537042b7f7f3ceb00a"} Jul 16 21:17:58.589680 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.589605 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6aed38fcf7a27a35333fc321f48b08db017400d4e22ce1537042b7f7f3ceb00a" Jul 16 21:17:58.728609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.728549 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" Jul 16 21:17:58.751125 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.751103 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" Jul 16 21:17:58.798397 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.798373 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2cbded0-5f0e-4865-a7d5-cd2619506273-bundle\") pod \"f2cbded0-5f0e-4865-a7d5-cd2619506273\" (UID: \"f2cbded0-5f0e-4865-a7d5-cd2619506273\") " Jul 16 21:17:58.798542 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.798426 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92841e70-433a-42e1-8313-5419ab6f6399-util\") pod \"92841e70-433a-42e1-8313-5419ab6f6399\" (UID: \"92841e70-433a-42e1-8313-5419ab6f6399\") " Jul 16 21:17:58.798542 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.798483 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92841e70-433a-42e1-8313-5419ab6f6399-bundle\") pod \"92841e70-433a-42e1-8313-5419ab6f6399\" (UID: \"92841e70-433a-42e1-8313-5419ab6f6399\") " Jul 16 21:17:58.798542 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.798504 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2cbded0-5f0e-4865-a7d5-cd2619506273-util\") pod \"f2cbded0-5f0e-4865-a7d5-cd2619506273\" (UID: \"f2cbded0-5f0e-4865-a7d5-cd2619506273\") " Jul 16 21:17:58.798648 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.798543 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk6hd\" (UniqueName: \"kubernetes.io/projected/f2cbded0-5f0e-4865-a7d5-cd2619506273-kube-api-access-wk6hd\") pod \"f2cbded0-5f0e-4865-a7d5-cd2619506273\" (UID: \"f2cbded0-5f0e-4865-a7d5-cd2619506273\") " Jul 16 21:17:58.798648 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.798624 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7dfz\" (UniqueName: \"kubernetes.io/projected/92841e70-433a-42e1-8313-5419ab6f6399-kube-api-access-f7dfz\") pod \"92841e70-433a-42e1-8313-5419ab6f6399\" (UID: \"92841e70-433a-42e1-8313-5419ab6f6399\") " Jul 16 21:17:58.798968 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.798938 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cbded0-5f0e-4865-a7d5-cd2619506273-bundle" (OuterVolumeSpecName: "bundle") pod "f2cbded0-5f0e-4865-a7d5-cd2619506273" (UID: "f2cbded0-5f0e-4865-a7d5-cd2619506273"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:17:58.799155 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.799083 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92841e70-433a-42e1-8313-5419ab6f6399-bundle" (OuterVolumeSpecName: "bundle") pod "92841e70-433a-42e1-8313-5419ab6f6399" (UID: "92841e70-433a-42e1-8313-5419ab6f6399"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:17:58.800784 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.800755 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cbded0-5f0e-4865-a7d5-cd2619506273-kube-api-access-wk6hd" (OuterVolumeSpecName: "kube-api-access-wk6hd") pod "f2cbded0-5f0e-4865-a7d5-cd2619506273" (UID: "f2cbded0-5f0e-4865-a7d5-cd2619506273"). InnerVolumeSpecName "kube-api-access-wk6hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:17:58.800881 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.800804 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92841e70-433a-42e1-8313-5419ab6f6399-kube-api-access-f7dfz" (OuterVolumeSpecName: "kube-api-access-f7dfz") pod "92841e70-433a-42e1-8313-5419ab6f6399" (UID: "92841e70-433a-42e1-8313-5419ab6f6399"). InnerVolumeSpecName "kube-api-access-f7dfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:17:58.804778 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.804756 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cbded0-5f0e-4865-a7d5-cd2619506273-util" (OuterVolumeSpecName: "util") pod "f2cbded0-5f0e-4865-a7d5-cd2619506273" (UID: "f2cbded0-5f0e-4865-a7d5-cd2619506273"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:17:58.804846 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.804815 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92841e70-433a-42e1-8313-5419ab6f6399-util" (OuterVolumeSpecName: "util") pod "92841e70-433a-42e1-8313-5419ab6f6399" (UID: "92841e70-433a-42e1-8313-5419ab6f6399"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:17:58.899897 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.899863 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92841e70-433a-42e1-8313-5419ab6f6399-bundle\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:17:58.899897 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.899891 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2cbded0-5f0e-4865-a7d5-cd2619506273-util\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:17:58.899897 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.899901 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk6hd\" (UniqueName: \"kubernetes.io/projected/f2cbded0-5f0e-4865-a7d5-cd2619506273-kube-api-access-wk6hd\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:17:58.900101 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.899911 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7dfz\" (UniqueName: \"kubernetes.io/projected/92841e70-433a-42e1-8313-5419ab6f6399-kube-api-access-f7dfz\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:17:58.900101 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.899921 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2cbded0-5f0e-4865-a7d5-cd2619506273-bundle\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:17:58.900101 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:58.899928 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92841e70-433a-42e1-8313-5419ab6f6399-util\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:17:59.596577 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:59.596547 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" Jul 16 21:17:59.596987 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:59.596575 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw" event={"ID":"f2cbded0-5f0e-4865-a7d5-cd2619506273","Type":"ContainerDied","Data":"1cd6f3254610f394415bc7ef43cdebfa4d23698f8ddd0d985ddd95217e062be6"} Jul 16 21:17:59.596987 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:59.596622 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cd6f3254610f394415bc7ef43cdebfa4d23698f8ddd0d985ddd95217e062be6" Jul 16 21:17:59.598510 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:59.598488 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" event={"ID":"92841e70-433a-42e1-8313-5419ab6f6399","Type":"ContainerDied","Data":"ec5ca480c1a8e027e3373ea467833323cbc878c8c34f199c91f4abce9c975b04"} Jul 16 21:17:59.598510 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:59.598506 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr" Jul 16 21:17:59.598615 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:17:59.598530 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec5ca480c1a8e027e3373ea467833323cbc878c8c34f199c91f4abce9c975b04" Jul 16 21:18:05.006021 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.005960 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-546ddb9b4f-zxhzc" podUID="92cbe129-7fcc-4674-956e-1930031c0ff2" containerName="console" containerID="cri-o://cfad0f14de7952ad7e4eaeed3f321defa77ffe2abad64c10334d844b8858087a" gracePeriod=15 Jul 16 21:18:05.250434 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.250412 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:18:05.354350 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.354267 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-trusted-ca-bundle\") pod \"92cbe129-7fcc-4674-956e-1930031c0ff2\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " Jul 16 21:18:05.354350 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.354325 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92cbe129-7fcc-4674-956e-1930031c0ff2-console-oauth-config\") pod \"92cbe129-7fcc-4674-956e-1930031c0ff2\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " Jul 16 21:18:05.354561 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.354356 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92cbe129-7fcc-4674-956e-1930031c0ff2-console-serving-cert\") pod \"92cbe129-7fcc-4674-956e-1930031c0ff2\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " Jul 16 21:18:05.354561 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.354395 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-service-ca\") pod \"92cbe129-7fcc-4674-956e-1930031c0ff2\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " Jul 16 21:18:05.354671 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.354558 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqtn2\" (UniqueName: \"kubernetes.io/projected/92cbe129-7fcc-4674-956e-1930031c0ff2-kube-api-access-tqtn2\") pod \"92cbe129-7fcc-4674-956e-1930031c0ff2\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " Jul 16 21:18:05.354671 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.354623 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-console-config\") pod \"92cbe129-7fcc-4674-956e-1930031c0ff2\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " Jul 16 21:18:05.354671 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.354651 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-oauth-serving-cert\") pod \"92cbe129-7fcc-4674-956e-1930031c0ff2\" (UID: \"92cbe129-7fcc-4674-956e-1930031c0ff2\") " Jul 16 21:18:05.354840 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.354790 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "92cbe129-7fcc-4674-956e-1930031c0ff2" (UID: "92cbe129-7fcc-4674-956e-1930031c0ff2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:18:05.355050 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.354962 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-trusted-ca-bundle\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:18:05.355050 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.355015 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-service-ca" (OuterVolumeSpecName: "service-ca") pod "92cbe129-7fcc-4674-956e-1930031c0ff2" (UID: "92cbe129-7fcc-4674-956e-1930031c0ff2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:18:05.355274 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.355245 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-console-config" (OuterVolumeSpecName: "console-config") pod "92cbe129-7fcc-4674-956e-1930031c0ff2" (UID: "92cbe129-7fcc-4674-956e-1930031c0ff2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:18:05.355421 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.355399 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "92cbe129-7fcc-4674-956e-1930031c0ff2" (UID: "92cbe129-7fcc-4674-956e-1930031c0ff2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:18:05.356768 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.356742 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92cbe129-7fcc-4674-956e-1930031c0ff2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "92cbe129-7fcc-4674-956e-1930031c0ff2" (UID: "92cbe129-7fcc-4674-956e-1930031c0ff2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:18:05.356903 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.356883 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92cbe129-7fcc-4674-956e-1930031c0ff2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "92cbe129-7fcc-4674-956e-1930031c0ff2" (UID: "92cbe129-7fcc-4674-956e-1930031c0ff2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:18:05.357160 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.357135 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92cbe129-7fcc-4674-956e-1930031c0ff2-kube-api-access-tqtn2" (OuterVolumeSpecName: "kube-api-access-tqtn2") pod "92cbe129-7fcc-4674-956e-1930031c0ff2" (UID: "92cbe129-7fcc-4674-956e-1930031c0ff2"). InnerVolumeSpecName "kube-api-access-tqtn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:18:05.455956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.455929 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqtn2\" (UniqueName: \"kubernetes.io/projected/92cbe129-7fcc-4674-956e-1930031c0ff2-kube-api-access-tqtn2\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:18:05.455956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.455954 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-console-config\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:18:05.455956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.455964 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-oauth-serving-cert\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:18:05.456222 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.455974 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92cbe129-7fcc-4674-956e-1930031c0ff2-console-oauth-config\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:18:05.456222 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.455983 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92cbe129-7fcc-4674-956e-1930031c0ff2-console-serving-cert\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:18:05.456222 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.455994 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92cbe129-7fcc-4674-956e-1930031c0ff2-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:18:05.628870 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.628842 2577 generic.go:356] "Generic (PLEG): container finished" podID="92cbe129-7fcc-4674-956e-1930031c0ff2" containerID="cfad0f14de7952ad7e4eaeed3f321defa77ffe2abad64c10334d844b8858087a" exitCode=0 Jul 16 21:18:05.629010 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.628889 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546ddb9b4f-zxhzc" Jul 16 21:18:05.629010 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.628897 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546ddb9b4f-zxhzc" event={"ID":"92cbe129-7fcc-4674-956e-1930031c0ff2","Type":"ContainerDied","Data":"cfad0f14de7952ad7e4eaeed3f321defa77ffe2abad64c10334d844b8858087a"} Jul 16 21:18:05.629010 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.628935 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546ddb9b4f-zxhzc" event={"ID":"92cbe129-7fcc-4674-956e-1930031c0ff2","Type":"ContainerDied","Data":"cdde7e7fe040003e2a4f34371d68883faa4c62d65617d1d02980eff8f2412da8"} Jul 16 21:18:05.629010 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.628954 2577 scope.go:117] "RemoveContainer" containerID="cfad0f14de7952ad7e4eaeed3f321defa77ffe2abad64c10334d844b8858087a" Jul 16 21:18:05.639314 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.639294 2577 scope.go:117] "RemoveContainer" containerID="cfad0f14de7952ad7e4eaeed3f321defa77ffe2abad64c10334d844b8858087a" Jul 16 21:18:05.639608 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:18:05.639582 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfad0f14de7952ad7e4eaeed3f321defa77ffe2abad64c10334d844b8858087a\": container with ID starting with cfad0f14de7952ad7e4eaeed3f321defa77ffe2abad64c10334d844b8858087a not found: ID does not exist" containerID="cfad0f14de7952ad7e4eaeed3f321defa77ffe2abad64c10334d844b8858087a" Jul 16 21:18:05.639709 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.639621 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfad0f14de7952ad7e4eaeed3f321defa77ffe2abad64c10334d844b8858087a"} err="failed to get container status \"cfad0f14de7952ad7e4eaeed3f321defa77ffe2abad64c10334d844b8858087a\": rpc error: code = NotFound desc = could not find container \"cfad0f14de7952ad7e4eaeed3f321defa77ffe2abad64c10334d844b8858087a\": container with ID starting with cfad0f14de7952ad7e4eaeed3f321defa77ffe2abad64c10334d844b8858087a not found: ID does not exist" Jul 16 21:18:05.667864 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.667837 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-546ddb9b4f-zxhzc"] Jul 16 21:18:05.672339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:05.672315 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-546ddb9b4f-zxhzc"] Jul 16 21:18:06.450847 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:06.450814 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92cbe129-7fcc-4674-956e-1930031c0ff2" path="/var/lib/kubelet/pods/92cbe129-7fcc-4674-956e-1930031c0ff2/volumes" Jul 16 21:18:33.009359 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009324 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7khc2"] Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009715 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92cbe129-7fcc-4674-956e-1930031c0ff2" containerName="console" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009730 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="92cbe129-7fcc-4674-956e-1930031c0ff2" containerName="console" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009744 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec078ff-b939-4e9a-a206-81ea686b278b" containerName="util" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009753 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec078ff-b939-4e9a-a206-81ea686b278b" containerName="util" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009768 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92841e70-433a-42e1-8313-5419ab6f6399" containerName="util" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009777 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="92841e70-433a-42e1-8313-5419ab6f6399" containerName="util" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009792 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ee8cd36-2859-4bf3-a453-e468eb83048d" containerName="util" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009800 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ee8cd36-2859-4bf3-a453-e468eb83048d" containerName="util" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009809 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ee8cd36-2859-4bf3-a453-e468eb83048d" containerName="extract" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009818 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ee8cd36-2859-4bf3-a453-e468eb83048d" containerName="extract" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009831 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec078ff-b939-4e9a-a206-81ea686b278b" containerName="extract" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009839 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec078ff-b939-4e9a-a206-81ea686b278b" containerName="extract" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009858 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92841e70-433a-42e1-8313-5419ab6f6399" containerName="pull" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009866 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="92841e70-433a-42e1-8313-5419ab6f6399" containerName="pull" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009876 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92841e70-433a-42e1-8313-5419ab6f6399" containerName="extract" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009884 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="92841e70-433a-42e1-8313-5419ab6f6399" containerName="extract" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009894 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2cbded0-5f0e-4865-a7d5-cd2619506273" containerName="extract" Jul 16 21:18:33.009898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009902 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2cbded0-5f0e-4865-a7d5-cd2619506273" containerName="extract" Jul 16 21:18:33.010661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009915 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2cbded0-5f0e-4865-a7d5-cd2619506273" containerName="util" Jul 16 21:18:33.010661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009924 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2cbded0-5f0e-4865-a7d5-cd2619506273" containerName="util" Jul 16 21:18:33.010661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009931 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2cbded0-5f0e-4865-a7d5-cd2619506273" containerName="pull" Jul 16 21:18:33.010661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009939 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2cbded0-5f0e-4865-a7d5-cd2619506273" containerName="pull" Jul 16 21:18:33.010661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009951 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ee8cd36-2859-4bf3-a453-e468eb83048d" containerName="pull" Jul 16 21:18:33.010661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009958 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ee8cd36-2859-4bf3-a453-e468eb83048d" containerName="pull" Jul 16 21:18:33.010661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009968 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec078ff-b939-4e9a-a206-81ea686b278b" containerName="pull" Jul 16 21:18:33.010661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.009973 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec078ff-b939-4e9a-a206-81ea686b278b" containerName="pull" Jul 16 21:18:33.010661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.010057 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ee8cd36-2859-4bf3-a453-e468eb83048d" containerName="extract" Jul 16 21:18:33.010661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.010071 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ec078ff-b939-4e9a-a206-81ea686b278b" containerName="extract" Jul 16 21:18:33.010661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.010081 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2cbded0-5f0e-4865-a7d5-cd2619506273" containerName="extract" Jul 16 21:18:33.010661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.010092 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="92841e70-433a-42e1-8313-5419ab6f6399" containerName="extract" Jul 16 21:18:33.010661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.010098 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="92cbe129-7fcc-4674-956e-1930031c0ff2" containerName="console" Jul 16 21:18:33.014727 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.014710 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7khc2" Jul 16 21:18:33.021023 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.020999 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 21:18:33.022104 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.022078 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-p8z8h\"" Jul 16 21:18:33.022216 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.022128 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 21:18:33.027563 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.027543 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7khc2"] Jul 16 21:18:33.096408 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.096385 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z2s2\" (UniqueName: \"kubernetes.io/projected/3c42ddfe-8200-4594-8c0f-12b3233d00e1-kube-api-access-4z2s2\") pod \"limitador-operator-controller-manager-859d7b78bd-7khc2\" (UID: \"3c42ddfe-8200-4594-8c0f-12b3233d00e1\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7khc2" Jul 16 21:18:33.197469 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.197440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2s2\" (UniqueName: \"kubernetes.io/projected/3c42ddfe-8200-4594-8c0f-12b3233d00e1-kube-api-access-4z2s2\") pod \"limitador-operator-controller-manager-859d7b78bd-7khc2\" (UID: \"3c42ddfe-8200-4594-8c0f-12b3233d00e1\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7khc2" Jul 16 21:18:33.213839 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.213814 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2s2\" (UniqueName: \"kubernetes.io/projected/3c42ddfe-8200-4594-8c0f-12b3233d00e1-kube-api-access-4z2s2\") pod \"limitador-operator-controller-manager-859d7b78bd-7khc2\" (UID: \"3c42ddfe-8200-4594-8c0f-12b3233d00e1\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7khc2" Jul 16 21:18:33.325538 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.325486 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7khc2" Jul 16 21:18:33.495754 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.495725 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7khc2"] Jul 16 21:18:33.498460 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:18:33.498433 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c42ddfe_8200_4594_8c0f_12b3233d00e1.slice/crio-2c25de9b5e06ecdf4ae6e54705b7894ebded85300cfd532ba7f1cf5bc171c3ca WatchSource:0}: Error finding container 2c25de9b5e06ecdf4ae6e54705b7894ebded85300cfd532ba7f1cf5bc171c3ca: Status 404 returned error can't find the container with id 2c25de9b5e06ecdf4ae6e54705b7894ebded85300cfd532ba7f1cf5bc171c3ca Jul 16 21:18:33.764752 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:33.764718 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7khc2" event={"ID":"3c42ddfe-8200-4594-8c0f-12b3233d00e1","Type":"ContainerStarted","Data":"2c25de9b5e06ecdf4ae6e54705b7894ebded85300cfd532ba7f1cf5bc171c3ca"} Jul 16 21:18:35.775997 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:35.775897 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7khc2" event={"ID":"3c42ddfe-8200-4594-8c0f-12b3233d00e1","Type":"ContainerStarted","Data":"679e169151d393fb41b8339ef9ae6ad1b26b94d2c74ed9f6bf1c89a415f7e759"} Jul 16 21:18:35.776384 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:35.776024 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7khc2" Jul 16 21:18:35.796124 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:35.796077 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7khc2" podStartSLOduration=1.7696761890000001 podStartE2EDuration="3.796062425s" totalImagesPullingTime="2.026386236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:18:35.795206641 +0000 UTC m=+531.970314850" watchObservedRunningTime="2026-07-16 21:18:35.796062425 +0000 UTC m=+531.971170620" Jul 16 21:18:46.783523 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:18:46.783494 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7khc2" Jul 16 21:19:44.266105 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:19:44.266071 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:19:44.268835 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:19:44.268810 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:20:58.017273 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.017234 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f"] Jul 16 21:20:58.019769 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.019751 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.023676 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.023648 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 21:20:58.023676 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.023663 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j767n\"" Jul 16 21:20:58.023887 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.023733 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 21:20:58.023887 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.023846 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 16 21:20:58.031884 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.031854 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f"] Jul 16 21:20:58.132203 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.132158 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb7eca7-64e2-4ab2-a119-053ccf18c111-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.132396 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.132274 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7zs7\" (UniqueName: \"kubernetes.io/projected/0cb7eca7-64e2-4ab2-a119-053ccf18c111-kube-api-access-b7zs7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.132396 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.132339 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb7eca7-64e2-4ab2-a119-053ccf18c111-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.132396 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.132372 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb7eca7-64e2-4ab2-a119-053ccf18c111-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.132562 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.132402 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb7eca7-64e2-4ab2-a119-053ccf18c111-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.132562 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.132497 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb7eca7-64e2-4ab2-a119-053ccf18c111-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.233764 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.233725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb7eca7-64e2-4ab2-a119-053ccf18c111-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.233959 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.233776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7zs7\" (UniqueName: \"kubernetes.io/projected/0cb7eca7-64e2-4ab2-a119-053ccf18c111-kube-api-access-b7zs7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.233959 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.233813 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb7eca7-64e2-4ab2-a119-053ccf18c111-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.233959 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.233848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb7eca7-64e2-4ab2-a119-053ccf18c111-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.233959 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.233887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb7eca7-64e2-4ab2-a119-053ccf18c111-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.233959 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.233937 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb7eca7-64e2-4ab2-a119-053ccf18c111-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.234333 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.234310 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb7eca7-64e2-4ab2-a119-053ccf18c111-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.234408 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.234347 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb7eca7-64e2-4ab2-a119-053ccf18c111-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.234408 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.234383 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb7eca7-64e2-4ab2-a119-053ccf18c111-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.236169 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.236146 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb7eca7-64e2-4ab2-a119-053ccf18c111-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.236367 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.236350 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb7eca7-64e2-4ab2-a119-053ccf18c111-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.262765 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.262723 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7zs7\" (UniqueName: \"kubernetes.io/projected/0cb7eca7-64e2-4ab2-a119-053ccf18c111-kube-api-access-b7zs7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jq76f\" (UID: \"0cb7eca7-64e2-4ab2-a119-053ccf18c111\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.330453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.330344 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:20:58.492893 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.492866 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f"] Jul 16 21:20:58.494754 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:20:58.494724 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb7eca7_64e2_4ab2_a119_053ccf18c111.slice/crio-daed3d3d7647ee2d38449527686d10c33f7fa9dbf444dc6a8a50fcde05bcb2b7 WatchSource:0}: Error finding container daed3d3d7647ee2d38449527686d10c33f7fa9dbf444dc6a8a50fcde05bcb2b7: Status 404 returned error can't find the container with id daed3d3d7647ee2d38449527686d10c33f7fa9dbf444dc6a8a50fcde05bcb2b7 Jul 16 21:20:58.496085 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:58.496068 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:20:59.491051 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:20:59.491013 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" event={"ID":"0cb7eca7-64e2-4ab2-a119-053ccf18c111","Type":"ContainerStarted","Data":"daed3d3d7647ee2d38449527686d10c33f7fa9dbf444dc6a8a50fcde05bcb2b7"} Jul 16 21:21:00.498174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:21:00.498131 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" event={"ID":"0cb7eca7-64e2-4ab2-a119-053ccf18c111","Type":"ContainerStarted","Data":"d97ac5ccc387535fc839a998cabd8425cd62271b6c9df6f98afd276eee6b136a"} Jul 16 21:21:00.518455 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:21:00.518400 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" podStartSLOduration=2.474196327 podStartE2EDuration="3.518384565s" totalImagesPullingTime="1.044188238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:20:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:21:00.516058477 +0000 UTC m=+676.691166671" watchObservedRunningTime="2026-07-16 21:21:00.518384565 +0000 UTC m=+676.693492799" Jul 16 21:21:08.330974 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:21:08.330931 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:21:08.331448 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:21:08.331252 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:21:08.343826 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:21:08.343801 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:21:08.549252 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:21:08.549224 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jq76f" Jul 16 21:24:44.301480 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:24:44.301450 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:24:44.301988 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:24:44.301608 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:29:44.335349 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:29:44.335316 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:29:44.335910 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:29:44.335734 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:34:44.370219 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:34:44.370164 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:34:44.371712 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:34:44.371686 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:37:19.719128 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.719094 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll"] Jul 16 21:37:19.722885 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.722864 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.727316 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.727296 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fc2328dc-openshift-default-dockercfg-d5cjr\"" Jul 16 21:37:19.759631 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.759601 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll"] Jul 16 21:37:19.808810 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.808780 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-workload-socket\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.808810 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.808811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-istio-envoy\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.809023 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.808830 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a12dfe-2221-40d0-87e0-49f86741d888-istiod-ca-cert\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.809023 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.808847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-istio-data\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.809023 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.808941 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-credential-socket\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.809023 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.808991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7f8x\" (UniqueName: \"kubernetes.io/projected/38a12dfe-2221-40d0-87e0-49f86741d888-kube-api-access-j7f8x\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.809023 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.809012 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a12dfe-2221-40d0-87e0-49f86741d888-istio-token\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.809211 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.809055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a12dfe-2221-40d0-87e0-49f86741d888-istio-podinfo\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.809211 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.809112 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-workload-certs\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.910527 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.910491 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7f8x\" (UniqueName: \"kubernetes.io/projected/38a12dfe-2221-40d0-87e0-49f86741d888-kube-api-access-j7f8x\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.910527 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.910533 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a12dfe-2221-40d0-87e0-49f86741d888-istio-token\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.910762 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.910564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a12dfe-2221-40d0-87e0-49f86741d888-istio-podinfo\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.910762 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.910608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-workload-certs\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.910762 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.910648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-workload-socket\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.910762 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.910671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-istio-envoy\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.910950 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.910752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a12dfe-2221-40d0-87e0-49f86741d888-istiod-ca-cert\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.910950 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.910799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-istio-data\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.910950 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.910857 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-credential-socket\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.911156 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.911099 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-workload-socket\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.911296 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.911229 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-workload-certs\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.911296 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.911287 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-credential-socket\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.911372 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.911299 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-istio-data\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.911573 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.911550 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a12dfe-2221-40d0-87e0-49f86741d888-istiod-ca-cert\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.913085 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.913068 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-istio-envoy\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.913364 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.913343 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a12dfe-2221-40d0-87e0-49f86741d888-istio-podinfo\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.920508 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.920481 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a12dfe-2221-40d0-87e0-49f86741d888-istio-token\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:19.920694 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:19.920674 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7f8x\" (UniqueName: \"kubernetes.io/projected/38a12dfe-2221-40d0-87e0-49f86741d888-kube-api-access-j7f8x\") pod \"e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:20.033751 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:20.033659 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:20.174882 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:20.174858 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll"] Jul 16 21:37:20.176576 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:37:20.176545 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a12dfe_2221_40d0_87e0_49f86741d888.slice/crio-fefde41dc1f2102d59d6219d5ab32a8c8fea243f71c44ebaf03f130925498bcd WatchSource:0}: Error finding container fefde41dc1f2102d59d6219d5ab32a8c8fea243f71c44ebaf03f130925498bcd: Status 404 returned error can't find the container with id fefde41dc1f2102d59d6219d5ab32a8c8fea243f71c44ebaf03f130925498bcd Jul 16 21:37:20.178938 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:20.178902 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:37:20.179037 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:20.178990 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:37:20.179082 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:20.179067 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:37:20.379665 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:20.379630 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" event={"ID":"38a12dfe-2221-40d0-87e0-49f86741d888","Type":"ContainerStarted","Data":"426c5e80050616af22eee916abaab44b92898aa6c1b277641d4b0af51a689f94"} Jul 16 21:37:20.379857 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:20.379693 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" event={"ID":"38a12dfe-2221-40d0-87e0-49f86741d888","Type":"ContainerStarted","Data":"fefde41dc1f2102d59d6219d5ab32a8c8fea243f71c44ebaf03f130925498bcd"} Jul 16 21:37:20.417302 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:20.416964 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" podStartSLOduration=1.416943997 podStartE2EDuration="1.416943997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:37:20.4152609 +0000 UTC m=+1656.590369112" watchObservedRunningTime="2026-07-16 21:37:20.416943997 +0000 UTC m=+1656.592052192" Jul 16 21:37:21.034757 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:21.034718 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:21.040330 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:21.040300 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:21.388732 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:21.388695 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:21.391968 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:21.391937 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:25.958936 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:25.958900 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk"] Jul 16 21:37:25.962637 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:25.962619 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" Jul 16 21:37:25.965387 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:25.965364 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:37:25.965520 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:25.965366 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:37:25.969956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:25.969931 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk"] Jul 16 21:37:26.074172 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:26.074130 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b8c996-7f9a-4a71-b0f2-d585a8c0af73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk\" (UID: \"80b8c996-7f9a-4a71-b0f2-d585a8c0af73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" Jul 16 21:37:26.174759 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:26.174717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b8c996-7f9a-4a71-b0f2-d585a8c0af73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk\" (UID: \"80b8c996-7f9a-4a71-b0f2-d585a8c0af73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" Jul 16 21:37:26.175468 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:26.175447 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b8c996-7f9a-4a71-b0f2-d585a8c0af73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk\" (UID: \"80b8c996-7f9a-4a71-b0f2-d585a8c0af73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" Jul 16 21:37:26.275682 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:26.275596 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" Jul 16 21:37:26.453554 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:26.453532 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk"] Jul 16 21:37:26.455632 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:37:26.455607 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b8c996_7f9a_4a71_b0f2_d585a8c0af73.slice/crio-f267fd42a053076506b190a90bac47152d93a4f9f2ed87aa4b1ab488aa66f87d WatchSource:0}: Error finding container f267fd42a053076506b190a90bac47152d93a4f9f2ed87aa4b1ab488aa66f87d: Status 404 returned error can't find the container with id f267fd42a053076506b190a90bac47152d93a4f9f2ed87aa4b1ab488aa66f87d Jul 16 21:37:26.457821 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:26.457804 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:37:27.421012 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:27.420941 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" event={"ID":"80b8c996-7f9a-4a71-b0f2-d585a8c0af73","Type":"ContainerStarted","Data":"f267fd42a053076506b190a90bac47152d93a4f9f2ed87aa4b1ab488aa66f87d"} Jul 16 21:37:29.435621 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:29.435582 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" event={"ID":"80b8c996-7f9a-4a71-b0f2-d585a8c0af73","Type":"ContainerStarted","Data":"6e2450c1ed9b995323ff7a3e48634d61d21148ddc11351ac73674159eca7792f"} Jul 16 21:37:29.454520 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:29.454472 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" podStartSLOduration=1.94684452 podStartE2EDuration="4.454457113s" totalImagesPullingTime="2.507612593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:37:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:37:29.452444268 +0000 UTC m=+1665.627552462" watchObservedRunningTime="2026-07-16 21:37:29.454457113 +0000 UTC m=+1665.629565305" Jul 16 21:37:30.443362 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:30.443326 2577 generic.go:356] "Generic (PLEG): container finished" podID="80b8c996-7f9a-4a71-b0f2-d585a8c0af73" containerID="6e2450c1ed9b995323ff7a3e48634d61d21148ddc11351ac73674159eca7792f" exitCode=7 Jul 16 21:37:30.443722 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:30.443388 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" event={"ID":"80b8c996-7f9a-4a71-b0f2-d585a8c0af73","Type":"ContainerDied","Data":"6e2450c1ed9b995323ff7a3e48634d61d21148ddc11351ac73674159eca7792f"} Jul 16 21:37:30.443722 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:30.443626 2577 scope.go:117] "RemoveContainer" containerID="6e2450c1ed9b995323ff7a3e48634d61d21148ddc11351ac73674159eca7792f" Jul 16 21:37:31.450061 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:31.450020 2577 generic.go:356] "Generic (PLEG): container finished" podID="80b8c996-7f9a-4a71-b0f2-d585a8c0af73" containerID="e05adadfde14f5cb08fb1ffa5bcd10634aa4914d7020531f11d34d658086971a" exitCode=7 Jul 16 21:37:31.450496 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:31.450097 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" event={"ID":"80b8c996-7f9a-4a71-b0f2-d585a8c0af73","Type":"ContainerDied","Data":"e05adadfde14f5cb08fb1ffa5bcd10634aa4914d7020531f11d34d658086971a"} Jul 16 21:37:31.450496 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:31.450152 2577 scope.go:117] "RemoveContainer" containerID="6e2450c1ed9b995323ff7a3e48634d61d21148ddc11351ac73674159eca7792f" Jul 16 21:37:31.450496 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:31.450403 2577 scope.go:117] "RemoveContainer" containerID="e05adadfde14f5cb08fb1ffa5bcd10634aa4914d7020531f11d34d658086971a" Jul 16 21:37:31.450691 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:37:31.450664 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk_odh-ai-gateway-infra(80b8c996-7f9a-4a71-b0f2-d585a8c0af73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" podUID="80b8c996-7f9a-4a71-b0f2-d585a8c0af73" Jul 16 21:37:32.456981 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:32.456947 2577 scope.go:117] "RemoveContainer" containerID="e05adadfde14f5cb08fb1ffa5bcd10634aa4914d7020531f11d34d658086971a" Jul 16 21:37:32.457368 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:37:32.457149 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk_odh-ai-gateway-infra(80b8c996-7f9a-4a71-b0f2-d585a8c0af73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" podUID="80b8c996-7f9a-4a71-b0f2-d585a8c0af73" Jul 16 21:37:33.463234 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:33.463199 2577 generic.go:356] "Generic (PLEG): container finished" podID="38a12dfe-2221-40d0-87e0-49f86741d888" containerID="426c5e80050616af22eee916abaab44b92898aa6c1b277641d4b0af51a689f94" exitCode=0 Jul 16 21:37:33.463639 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:33.463212 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" event={"ID":"38a12dfe-2221-40d0-87e0-49f86741d888","Type":"ContainerDied","Data":"426c5e80050616af22eee916abaab44b92898aa6c1b277641d4b0af51a689f94"} Jul 16 21:37:33.463639 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:33.463615 2577 scope.go:117] "RemoveContainer" containerID="426c5e80050616af22eee916abaab44b92898aa6c1b277641d4b0af51a689f94" Jul 16 21:37:33.464585 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:33.464552 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:37:33.464690 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:33.464621 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:37:33.464690 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:33.464654 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:37:34.472693 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:34.472660 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" event={"ID":"38a12dfe-2221-40d0-87e0-49f86741d888","Type":"ContainerStarted","Data":"4f934b225de0d0df20481699fd931a7ad38644694b5a843094276c2a8d2014c4"} Jul 16 21:37:35.034491 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:35.034457 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:35.035674 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:35.035649 2577 patch_prober.go:28] interesting pod/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 16 21:37:35.035786 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:35.035701 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" podUID="38a12dfe-2221-40d0-87e0-49f86741d888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 16 21:37:36.034842 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:36.034810 2577 patch_prober.go:28] interesting pod/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 16 21:37:36.035271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:36.034875 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" podUID="38a12dfe-2221-40d0-87e0-49f86741d888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 16 21:37:36.388864 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:36.388775 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:37.034161 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:37.034130 2577 patch_prober.go:28] interesting pod/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 16 21:37:37.034342 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:37.034205 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" podUID="38a12dfe-2221-40d0-87e0-49f86741d888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 16 21:37:38.034854 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:38.034817 2577 patch_prober.go:28] interesting pod/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 16 21:37:38.035258 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:38.034880 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" podUID="38a12dfe-2221-40d0-87e0-49f86741d888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 16 21:37:39.038339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:39.038310 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:39.039407 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:39.039383 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:37:44.447486 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:44.447458 2577 scope.go:117] "RemoveContainer" containerID="e05adadfde14f5cb08fb1ffa5bcd10634aa4914d7020531f11d34d658086971a" Jul 16 21:37:45.541095 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:45.541060 2577 generic.go:356] "Generic (PLEG): container finished" podID="80b8c996-7f9a-4a71-b0f2-d585a8c0af73" containerID="8b86aea77b6c2f3e12d63c2fa44f2ee90827c98e62090fc3d3ac7462803eb83d" exitCode=7 Jul 16 21:37:45.541553 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:45.541136 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" event={"ID":"80b8c996-7f9a-4a71-b0f2-d585a8c0af73","Type":"ContainerDied","Data":"8b86aea77b6c2f3e12d63c2fa44f2ee90827c98e62090fc3d3ac7462803eb83d"} Jul 16 21:37:45.541553 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:45.541232 2577 scope.go:117] "RemoveContainer" containerID="e05adadfde14f5cb08fb1ffa5bcd10634aa4914d7020531f11d34d658086971a" Jul 16 21:37:45.541553 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:45.541550 2577 scope.go:117] "RemoveContainer" containerID="8b86aea77b6c2f3e12d63c2fa44f2ee90827c98e62090fc3d3ac7462803eb83d" Jul 16 21:37:45.541788 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:37:45.541771 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk_odh-ai-gateway-infra(80b8c996-7f9a-4a71-b0f2-d585a8c0af73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" podUID="80b8c996-7f9a-4a71-b0f2-d585a8c0af73" Jul 16 21:37:46.569266 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:46.569234 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk"] Jul 16 21:37:46.698878 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:46.698854 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" Jul 16 21:37:46.776012 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:46.775975 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b8c996-7f9a-4a71-b0f2-d585a8c0af73-maas-api-service-ca\") pod \"80b8c996-7f9a-4a71-b0f2-d585a8c0af73\" (UID: \"80b8c996-7f9a-4a71-b0f2-d585a8c0af73\") " Jul 16 21:37:46.776361 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:46.776336 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b8c996-7f9a-4a71-b0f2-d585a8c0af73-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80b8c996-7f9a-4a71-b0f2-d585a8c0af73" (UID: "80b8c996-7f9a-4a71-b0f2-d585a8c0af73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:37:46.877112 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:46.877085 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b8c996-7f9a-4a71-b0f2-d585a8c0af73-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:37:47.554537 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:47.554507 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" Jul 16 21:37:47.554734 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:47.554548 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk" event={"ID":"80b8c996-7f9a-4a71-b0f2-d585a8c0af73","Type":"ContainerDied","Data":"f267fd42a053076506b190a90bac47152d93a4f9f2ed87aa4b1ab488aa66f87d"} Jul 16 21:37:47.554734 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:47.554605 2577 scope.go:117] "RemoveContainer" containerID="8b86aea77b6c2f3e12d63c2fa44f2ee90827c98e62090fc3d3ac7462803eb83d" Jul 16 21:37:47.577091 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:47.577067 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk"] Jul 16 21:37:47.583667 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:47.583646 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-pxxsk"] Jul 16 21:37:48.451109 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:37:48.451079 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80b8c996-7f9a-4a71-b0f2-d585a8c0af73" path="/var/lib/kubelet/pods/80b8c996-7f9a-4a71-b0f2-d585a8c0af73/volumes" Jul 16 21:38:26.043412 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.043334 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7"] Jul 16 21:38:26.043912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.043759 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b8c996-7f9a-4a71-b0f2-d585a8c0af73" containerName="revoke-keys" Jul 16 21:38:26.043912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.043771 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b8c996-7f9a-4a71-b0f2-d585a8c0af73" containerName="revoke-keys" Jul 16 21:38:26.043912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.043779 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b8c996-7f9a-4a71-b0f2-d585a8c0af73" containerName="revoke-keys" Jul 16 21:38:26.043912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.043786 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b8c996-7f9a-4a71-b0f2-d585a8c0af73" containerName="revoke-keys" Jul 16 21:38:26.043912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.043794 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b8c996-7f9a-4a71-b0f2-d585a8c0af73" containerName="revoke-keys" Jul 16 21:38:26.043912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.043804 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b8c996-7f9a-4a71-b0f2-d585a8c0af73" containerName="revoke-keys" Jul 16 21:38:26.043912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.043866 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b8c996-7f9a-4a71-b0f2-d585a8c0af73" containerName="revoke-keys" Jul 16 21:38:26.043912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.043875 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b8c996-7f9a-4a71-b0f2-d585a8c0af73" containerName="revoke-keys" Jul 16 21:38:26.043912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.043881 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b8c996-7f9a-4a71-b0f2-d585a8c0af73" containerName="revoke-keys" Jul 16 21:38:26.046764 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.046750 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" Jul 16 21:38:26.051864 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.051838 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:38:26.051992 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.051962 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:38:26.066605 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.066581 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7"] Jul 16 21:38:26.141071 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.141031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/121b0e51-f1c8-4cd8-93f9-7c077b697cce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7\" (UID: \"121b0e51-f1c8-4cd8-93f9-7c077b697cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" Jul 16 21:38:26.241547 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.241512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/121b0e51-f1c8-4cd8-93f9-7c077b697cce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7\" (UID: \"121b0e51-f1c8-4cd8-93f9-7c077b697cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" Jul 16 21:38:26.242122 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.242100 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/121b0e51-f1c8-4cd8-93f9-7c077b697cce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7\" (UID: \"121b0e51-f1c8-4cd8-93f9-7c077b697cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" Jul 16 21:38:26.356916 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.356835 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" Jul 16 21:38:26.488610 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.488579 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7"] Jul 16 21:38:26.490434 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:38:26.490405 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121b0e51_f1c8_4cd8_93f9_7c077b697cce.slice/crio-0f9f770d4eaa9383944be98b788e399b7441f466ac49b165555cac95ff5959a9 WatchSource:0}: Error finding container 0f9f770d4eaa9383944be98b788e399b7441f466ac49b165555cac95ff5959a9: Status 404 returned error can't find the container with id 0f9f770d4eaa9383944be98b788e399b7441f466ac49b165555cac95ff5959a9 Jul 16 21:38:26.754332 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:26.754300 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" event={"ID":"121b0e51-f1c8-4cd8-93f9-7c077b697cce","Type":"ContainerStarted","Data":"0f9f770d4eaa9383944be98b788e399b7441f466ac49b165555cac95ff5959a9"} Jul 16 21:38:27.760730 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:27.760696 2577 generic.go:356] "Generic (PLEG): container finished" podID="121b0e51-f1c8-4cd8-93f9-7c077b697cce" containerID="7df14cb75f35bdb0ff5e68290fb8bc34e7dca0f9de8b8135376a806982e89cca" exitCode=7 Jul 16 21:38:27.761129 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:27.760779 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" event={"ID":"121b0e51-f1c8-4cd8-93f9-7c077b697cce","Type":"ContainerDied","Data":"7df14cb75f35bdb0ff5e68290fb8bc34e7dca0f9de8b8135376a806982e89cca"} Jul 16 21:38:27.761129 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:27.760932 2577 scope.go:117] "RemoveContainer" containerID="7df14cb75f35bdb0ff5e68290fb8bc34e7dca0f9de8b8135376a806982e89cca" Jul 16 21:38:28.767824 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:28.767737 2577 generic.go:356] "Generic (PLEG): container finished" podID="121b0e51-f1c8-4cd8-93f9-7c077b697cce" containerID="1f5c83a1ea3ed95cc45003b9697c19f349740c921857e7c1d28133c20e0afb74" exitCode=7 Jul 16 21:38:28.767824 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:28.767814 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" event={"ID":"121b0e51-f1c8-4cd8-93f9-7c077b697cce","Type":"ContainerDied","Data":"1f5c83a1ea3ed95cc45003b9697c19f349740c921857e7c1d28133c20e0afb74"} Jul 16 21:38:28.768245 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:28.767861 2577 scope.go:117] "RemoveContainer" containerID="7df14cb75f35bdb0ff5e68290fb8bc34e7dca0f9de8b8135376a806982e89cca" Jul 16 21:38:28.768245 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:28.768111 2577 scope.go:117] "RemoveContainer" containerID="1f5c83a1ea3ed95cc45003b9697c19f349740c921857e7c1d28133c20e0afb74" Jul 16 21:38:28.768369 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:38:28.768349 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7_odh-ai-gateway-infra(121b0e51-f1c8-4cd8-93f9-7c077b697cce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" podUID="121b0e51-f1c8-4cd8-93f9-7c077b697cce" Jul 16 21:38:29.774110 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:29.774081 2577 scope.go:117] "RemoveContainer" containerID="1f5c83a1ea3ed95cc45003b9697c19f349740c921857e7c1d28133c20e0afb74" Jul 16 21:38:29.774592 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:38:29.774351 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7_odh-ai-gateway-infra(121b0e51-f1c8-4cd8-93f9-7c077b697cce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" podUID="121b0e51-f1c8-4cd8-93f9-7c077b697cce" Jul 16 21:38:40.445822 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:40.445789 2577 scope.go:117] "RemoveContainer" containerID="1f5c83a1ea3ed95cc45003b9697c19f349740c921857e7c1d28133c20e0afb74" Jul 16 21:38:40.831698 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:40.831613 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" event={"ID":"121b0e51-f1c8-4cd8-93f9-7c077b697cce","Type":"ContainerStarted","Data":"0e82e06d8f4ded36b8c33efaa40eba163071619fdff351756505ca2588a5c6e8"} Jul 16 21:38:40.852888 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:40.852839 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" podStartSLOduration=14.852823562 podStartE2EDuration="14.852823562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:38:40.850098228 +0000 UTC m=+1737.025206417" watchObservedRunningTime="2026-07-16 21:38:40.852823562 +0000 UTC m=+1737.027931754" Jul 16 21:38:41.838297 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:41.838265 2577 generic.go:356] "Generic (PLEG): container finished" podID="121b0e51-f1c8-4cd8-93f9-7c077b697cce" containerID="0e82e06d8f4ded36b8c33efaa40eba163071619fdff351756505ca2588a5c6e8" exitCode=7 Jul 16 21:38:41.838703 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:41.838336 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" event={"ID":"121b0e51-f1c8-4cd8-93f9-7c077b697cce","Type":"ContainerDied","Data":"0e82e06d8f4ded36b8c33efaa40eba163071619fdff351756505ca2588a5c6e8"} Jul 16 21:38:41.838703 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:41.838395 2577 scope.go:117] "RemoveContainer" containerID="1f5c83a1ea3ed95cc45003b9697c19f349740c921857e7c1d28133c20e0afb74" Jul 16 21:38:41.838703 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:41.838664 2577 scope.go:117] "RemoveContainer" containerID="0e82e06d8f4ded36b8c33efaa40eba163071619fdff351756505ca2588a5c6e8" Jul 16 21:38:41.839027 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:38:41.838845 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7_odh-ai-gateway-infra(121b0e51-f1c8-4cd8-93f9-7c077b697cce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" podUID="121b0e51-f1c8-4cd8-93f9-7c077b697cce" Jul 16 21:38:41.873005 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:41.872975 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7"] Jul 16 21:38:42.972096 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:42.972069 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" Jul 16 21:38:43.087098 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:43.087066 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/121b0e51-f1c8-4cd8-93f9-7c077b697cce-maas-api-service-ca\") pod \"121b0e51-f1c8-4cd8-93f9-7c077b697cce\" (UID: \"121b0e51-f1c8-4cd8-93f9-7c077b697cce\") " Jul 16 21:38:43.087455 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:43.087432 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121b0e51-f1c8-4cd8-93f9-7c077b697cce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "121b0e51-f1c8-4cd8-93f9-7c077b697cce" (UID: "121b0e51-f1c8-4cd8-93f9-7c077b697cce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:38:43.188752 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:43.188716 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/121b0e51-f1c8-4cd8-93f9-7c077b697cce-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:38:43.850241 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:43.850209 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" Jul 16 21:38:43.850241 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:43.850209 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7" event={"ID":"121b0e51-f1c8-4cd8-93f9-7c077b697cce","Type":"ContainerDied","Data":"0f9f770d4eaa9383944be98b788e399b7441f466ac49b165555cac95ff5959a9"} Jul 16 21:38:43.850478 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:43.850276 2577 scope.go:117] "RemoveContainer" containerID="0e82e06d8f4ded36b8c33efaa40eba163071619fdff351756505ca2588a5c6e8" Jul 16 21:38:43.872459 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:43.872433 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7"] Jul 16 21:38:43.883696 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:43.878752 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xbxm7"] Jul 16 21:38:44.450931 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:38:44.450902 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="121b0e51-f1c8-4cd8-93f9-7c077b697cce" path="/var/lib/kubelet/pods/121b0e51-f1c8-4cd8-93f9-7c077b697cce/volumes" Jul 16 21:39:16.082751 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.082715 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5"] Jul 16 21:39:16.083263 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.083099 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="121b0e51-f1c8-4cd8-93f9-7c077b697cce" containerName="revoke-keys" Jul 16 21:39:16.083263 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.083112 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="121b0e51-f1c8-4cd8-93f9-7c077b697cce" containerName="revoke-keys" Jul 16 21:39:16.083263 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.083120 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="121b0e51-f1c8-4cd8-93f9-7c077b697cce" containerName="revoke-keys" Jul 16 21:39:16.083263 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.083125 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="121b0e51-f1c8-4cd8-93f9-7c077b697cce" containerName="revoke-keys" Jul 16 21:39:16.083263 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.083215 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="121b0e51-f1c8-4cd8-93f9-7c077b697cce" containerName="revoke-keys" Jul 16 21:39:16.083263 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.083226 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="121b0e51-f1c8-4cd8-93f9-7c077b697cce" containerName="revoke-keys" Jul 16 21:39:16.083263 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.083239 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="121b0e51-f1c8-4cd8-93f9-7c077b697cce" containerName="revoke-keys" Jul 16 21:39:16.086121 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.086104 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" Jul 16 21:39:16.088654 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.088632 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:39:16.088962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.088946 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:39:16.095306 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.095280 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5"] Jul 16 21:39:16.183114 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.183073 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c0adb5-3334-4ce0-a333-98f7b5f06837-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5\" (UID: \"17c0adb5-3334-4ce0-a333-98f7b5f06837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" Jul 16 21:39:16.283783 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.283752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c0adb5-3334-4ce0-a333-98f7b5f06837-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5\" (UID: \"17c0adb5-3334-4ce0-a333-98f7b5f06837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" Jul 16 21:39:16.284399 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.284379 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c0adb5-3334-4ce0-a333-98f7b5f06837-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5\" (UID: \"17c0adb5-3334-4ce0-a333-98f7b5f06837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" Jul 16 21:39:16.397521 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.397493 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" Jul 16 21:39:16.533286 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:16.533061 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5"] Jul 16 21:39:17.019089 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:17.018997 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" event={"ID":"17c0adb5-3334-4ce0-a333-98f7b5f06837","Type":"ContainerStarted","Data":"82ec0f69024dccb1c0d8000a34ce815d817f8c53ada735b26d2a9d2cd4a9e27c"} Jul 16 21:39:17.019089 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:17.019052 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" event={"ID":"17c0adb5-3334-4ce0-a333-98f7b5f06837","Type":"ContainerStarted","Data":"4c4779983afd66df035bc2cdb7790c6f851a55ae7db1b2bb8a37455495398c9b"} Jul 16 21:39:17.047573 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:17.047518 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" podStartSLOduration=1.047503736 podStartE2EDuration="1.047503736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:39:17.044331247 +0000 UTC m=+1773.219439440" watchObservedRunningTime="2026-07-16 21:39:17.047503736 +0000 UTC m=+1773.222611929" Jul 16 21:39:18.025576 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:18.025539 2577 generic.go:356] "Generic (PLEG): container finished" podID="17c0adb5-3334-4ce0-a333-98f7b5f06837" containerID="82ec0f69024dccb1c0d8000a34ce815d817f8c53ada735b26d2a9d2cd4a9e27c" exitCode=7 Jul 16 21:39:18.025975 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:18.025609 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" event={"ID":"17c0adb5-3334-4ce0-a333-98f7b5f06837","Type":"ContainerDied","Data":"82ec0f69024dccb1c0d8000a34ce815d817f8c53ada735b26d2a9d2cd4a9e27c"} Jul 16 21:39:18.025975 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:18.025784 2577 scope.go:117] "RemoveContainer" containerID="82ec0f69024dccb1c0d8000a34ce815d817f8c53ada735b26d2a9d2cd4a9e27c" Jul 16 21:39:19.032671 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:19.032633 2577 generic.go:356] "Generic (PLEG): container finished" podID="17c0adb5-3334-4ce0-a333-98f7b5f06837" containerID="083489ed714a6ec86bf205bb3d4c084d6eac30f0b8530e1e9a3f445b2468c365" exitCode=7 Jul 16 21:39:19.033079 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:19.032720 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" event={"ID":"17c0adb5-3334-4ce0-a333-98f7b5f06837","Type":"ContainerDied","Data":"083489ed714a6ec86bf205bb3d4c084d6eac30f0b8530e1e9a3f445b2468c365"} Jul 16 21:39:19.033079 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:19.032785 2577 scope.go:117] "RemoveContainer" containerID="82ec0f69024dccb1c0d8000a34ce815d817f8c53ada735b26d2a9d2cd4a9e27c" Jul 16 21:39:19.033079 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:19.033031 2577 scope.go:117] "RemoveContainer" containerID="083489ed714a6ec86bf205bb3d4c084d6eac30f0b8530e1e9a3f445b2468c365" Jul 16 21:39:19.033279 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:39:19.033266 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5_odh-ai-gateway-infra(17c0adb5-3334-4ce0-a333-98f7b5f06837)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" podUID="17c0adb5-3334-4ce0-a333-98f7b5f06837" Jul 16 21:39:20.040718 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:20.040689 2577 scope.go:117] "RemoveContainer" containerID="083489ed714a6ec86bf205bb3d4c084d6eac30f0b8530e1e9a3f445b2468c365" Jul 16 21:39:20.041133 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:39:20.040895 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5_odh-ai-gateway-infra(17c0adb5-3334-4ce0-a333-98f7b5f06837)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" podUID="17c0adb5-3334-4ce0-a333-98f7b5f06837" Jul 16 21:39:31.445640 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:31.445608 2577 scope.go:117] "RemoveContainer" containerID="083489ed714a6ec86bf205bb3d4c084d6eac30f0b8530e1e9a3f445b2468c365" Jul 16 21:39:32.102621 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:32.102586 2577 generic.go:356] "Generic (PLEG): container finished" podID="17c0adb5-3334-4ce0-a333-98f7b5f06837" containerID="145ba76306bf3d2d836d647fa15d3d5faea7222f4986eaa8ef056e526815295f" exitCode=7 Jul 16 21:39:32.102786 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:32.102654 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" event={"ID":"17c0adb5-3334-4ce0-a333-98f7b5f06837","Type":"ContainerDied","Data":"145ba76306bf3d2d836d647fa15d3d5faea7222f4986eaa8ef056e526815295f"} Jul 16 21:39:32.102786 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:32.102713 2577 scope.go:117] "RemoveContainer" containerID="083489ed714a6ec86bf205bb3d4c084d6eac30f0b8530e1e9a3f445b2468c365" Jul 16 21:39:32.103034 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:32.103014 2577 scope.go:117] "RemoveContainer" containerID="145ba76306bf3d2d836d647fa15d3d5faea7222f4986eaa8ef056e526815295f" Jul 16 21:39:32.103287 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:39:32.103262 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5_odh-ai-gateway-infra(17c0adb5-3334-4ce0-a333-98f7b5f06837)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" podUID="17c0adb5-3334-4ce0-a333-98f7b5f06837" Jul 16 21:39:33.142197 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:33.142160 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5"] Jul 16 21:39:33.277412 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:33.277382 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" Jul 16 21:39:33.334946 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:33.334914 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c0adb5-3334-4ce0-a333-98f7b5f06837-maas-api-service-ca\") pod \"17c0adb5-3334-4ce0-a333-98f7b5f06837\" (UID: \"17c0adb5-3334-4ce0-a333-98f7b5f06837\") " Jul 16 21:39:33.335332 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:33.335312 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c0adb5-3334-4ce0-a333-98f7b5f06837-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "17c0adb5-3334-4ce0-a333-98f7b5f06837" (UID: "17c0adb5-3334-4ce0-a333-98f7b5f06837"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:39:33.436142 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:33.436054 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c0adb5-3334-4ce0-a333-98f7b5f06837-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:39:34.116151 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:34.116123 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" Jul 16 21:39:34.116347 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:34.116122 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5" event={"ID":"17c0adb5-3334-4ce0-a333-98f7b5f06837","Type":"ContainerDied","Data":"4c4779983afd66df035bc2cdb7790c6f851a55ae7db1b2bb8a37455495398c9b"} Jul 16 21:39:34.116347 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:34.116253 2577 scope.go:117] "RemoveContainer" containerID="145ba76306bf3d2d836d647fa15d3d5faea7222f4986eaa8ef056e526815295f" Jul 16 21:39:34.142737 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:34.142705 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5"] Jul 16 21:39:34.146102 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:34.146077 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-f2pg5"] Jul 16 21:39:34.450179 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:34.450143 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17c0adb5-3334-4ce0-a333-98f7b5f06837" path="/var/lib/kubelet/pods/17c0adb5-3334-4ce0-a333-98f7b5f06837/volumes" Jul 16 21:39:44.404341 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:44.404312 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:39:44.407736 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:39:44.407712 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:40:06.120740 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.120702 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp"] Jul 16 21:40:06.121261 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.121059 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="121b0e51-f1c8-4cd8-93f9-7c077b697cce" containerName="revoke-keys" Jul 16 21:40:06.121261 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.121071 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="121b0e51-f1c8-4cd8-93f9-7c077b697cce" containerName="revoke-keys" Jul 16 21:40:06.121261 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.121085 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c0adb5-3334-4ce0-a333-98f7b5f06837" containerName="revoke-keys" Jul 16 21:40:06.121261 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.121090 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c0adb5-3334-4ce0-a333-98f7b5f06837" containerName="revoke-keys" Jul 16 21:40:06.121261 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.121097 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c0adb5-3334-4ce0-a333-98f7b5f06837" containerName="revoke-keys" Jul 16 21:40:06.121261 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.121103 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c0adb5-3334-4ce0-a333-98f7b5f06837" containerName="revoke-keys" Jul 16 21:40:06.121261 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.121113 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c0adb5-3334-4ce0-a333-98f7b5f06837" containerName="revoke-keys" Jul 16 21:40:06.121261 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.121118 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c0adb5-3334-4ce0-a333-98f7b5f06837" containerName="revoke-keys" Jul 16 21:40:06.121261 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.121173 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="17c0adb5-3334-4ce0-a333-98f7b5f06837" containerName="revoke-keys" Jul 16 21:40:06.121261 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.121193 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="17c0adb5-3334-4ce0-a333-98f7b5f06837" containerName="revoke-keys" Jul 16 21:40:06.124098 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.124082 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp" Jul 16 21:40:06.126816 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.126791 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:40:06.126816 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.126805 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:40:06.131786 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.131758 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp"] Jul 16 21:40:06.144536 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.144496 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ee0794-5103-4670-ab89-6a5129a64e3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp\" (UID: \"a9ee0794-5103-4670-ab89-6a5129a64e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp" Jul 16 21:40:06.245300 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.245264 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ee0794-5103-4670-ab89-6a5129a64e3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp\" (UID: \"a9ee0794-5103-4670-ab89-6a5129a64e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp" Jul 16 21:40:06.245949 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.245926 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ee0794-5103-4670-ab89-6a5129a64e3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp\" (UID: \"a9ee0794-5103-4670-ab89-6a5129a64e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp" Jul 16 21:40:06.435215 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.435163 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp" Jul 16 21:40:06.571439 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:06.571413 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp"] Jul 16 21:40:06.573408 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:40:06.573380 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ee0794_5103_4670_ab89_6a5129a64e3c.slice/crio-75c370d289b1fc0085ed575de0e16f06f09e2295476b0a5a36e8e6308828d088 WatchSource:0}: Error finding container 75c370d289b1fc0085ed575de0e16f06f09e2295476b0a5a36e8e6308828d088: Status 404 returned error can't find the container with id 75c370d289b1fc0085ed575de0e16f06f09e2295476b0a5a36e8e6308828d088 Jul 16 21:40:07.287924 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:07.287888 2577 generic.go:356] "Generic (PLEG): container finished" podID="a9ee0794-5103-4670-ab89-6a5129a64e3c" containerID="27ad5a6cb2f7ee0c7f6b3d41b06ce06bbb908945693dd5aa0e7b593933d5c27b" exitCode=7 Jul 16 21:40:07.288334 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:07.287949 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp" event={"ID":"a9ee0794-5103-4670-ab89-6a5129a64e3c","Type":"ContainerDied","Data":"27ad5a6cb2f7ee0c7f6b3d41b06ce06bbb908945693dd5aa0e7b593933d5c27b"} Jul 16 21:40:07.288334 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:07.287995 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp" event={"ID":"a9ee0794-5103-4670-ab89-6a5129a64e3c","Type":"ContainerStarted","Data":"75c370d289b1fc0085ed575de0e16f06f09e2295476b0a5a36e8e6308828d088"} Jul 16 21:40:07.288334 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:07.288239 2577 scope.go:117] "RemoveContainer" containerID="27ad5a6cb2f7ee0c7f6b3d41b06ce06bbb908945693dd5aa0e7b593933d5c27b" Jul 16 21:40:08.294815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:08.294726 2577 generic.go:356] "Generic (PLEG): container finished" podID="a9ee0794-5103-4670-ab89-6a5129a64e3c" containerID="0dd1d2b3de80cb09053b21c6d5dda6338529fbb190fd52a2ec919c97b716479e" exitCode=7 Jul 16 21:40:08.295302 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:08.294811 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp" event={"ID":"a9ee0794-5103-4670-ab89-6a5129a64e3c","Type":"ContainerDied","Data":"0dd1d2b3de80cb09053b21c6d5dda6338529fbb190fd52a2ec919c97b716479e"} Jul 16 21:40:08.295302 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:08.294871 2577 scope.go:117] "RemoveContainer" containerID="27ad5a6cb2f7ee0c7f6b3d41b06ce06bbb908945693dd5aa0e7b593933d5c27b" Jul 16 21:40:08.295302 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:08.295142 2577 scope.go:117] "RemoveContainer" containerID="0dd1d2b3de80cb09053b21c6d5dda6338529fbb190fd52a2ec919c97b716479e" Jul 16 21:40:08.295461 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:40:08.295419 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp_odh-ai-gateway-infra(a9ee0794-5103-4670-ab89-6a5129a64e3c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp" podUID="a9ee0794-5103-4670-ab89-6a5129a64e3c" Jul 16 21:40:09.301515 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:09.301486 2577 scope.go:117] "RemoveContainer" containerID="0dd1d2b3de80cb09053b21c6d5dda6338529fbb190fd52a2ec919c97b716479e" Jul 16 21:40:09.301901 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:40:09.301697 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp_odh-ai-gateway-infra(a9ee0794-5103-4670-ab89-6a5129a64e3c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp" podUID="a9ee0794-5103-4670-ab89-6a5129a64e3c" Jul 16 21:40:21.444716 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:21.444681 2577 scope.go:117] "RemoveContainer" containerID="0dd1d2b3de80cb09053b21c6d5dda6338529fbb190fd52a2ec919c97b716479e" Jul 16 21:40:22.369804 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:22.369715 2577 generic.go:356] "Generic (PLEG): container finished" podID="a9ee0794-5103-4670-ab89-6a5129a64e3c" containerID="2987078abd95e0d6a9824f692866b1a481fa3168ed1bc97221283a79141ec6dc" exitCode=7 Jul 16 21:40:22.369804 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:22.369788 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp" event={"ID":"a9ee0794-5103-4670-ab89-6a5129a64e3c","Type":"ContainerDied","Data":"2987078abd95e0d6a9824f692866b1a481fa3168ed1bc97221283a79141ec6dc"} Jul 16 21:40:22.369999 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:22.369848 2577 scope.go:117] "RemoveContainer" containerID="0dd1d2b3de80cb09053b21c6d5dda6338529fbb190fd52a2ec919c97b716479e" Jul 16 21:40:22.370196 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:22.370167 2577 scope.go:117] "RemoveContainer" containerID="2987078abd95e0d6a9824f692866b1a481fa3168ed1bc97221283a79141ec6dc" Jul 16 21:40:22.370463 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:40:22.370437 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp_odh-ai-gateway-infra(a9ee0794-5103-4670-ab89-6a5129a64e3c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp" podUID="a9ee0794-5103-4670-ab89-6a5129a64e3c" Jul 16 21:40:23.398800 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:23.398763 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp"] Jul 16 21:40:23.538759 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:23.538731 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp" Jul 16 21:40:23.605429 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:23.605394 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ee0794-5103-4670-ab89-6a5129a64e3c-maas-api-service-ca\") pod \"a9ee0794-5103-4670-ab89-6a5129a64e3c\" (UID: \"a9ee0794-5103-4670-ab89-6a5129a64e3c\") " Jul 16 21:40:23.605756 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:23.605734 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ee0794-5103-4670-ab89-6a5129a64e3c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9ee0794-5103-4670-ab89-6a5129a64e3c" (UID: "a9ee0794-5103-4670-ab89-6a5129a64e3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:40:23.706928 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:23.706846 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ee0794-5103-4670-ab89-6a5129a64e3c-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:40:24.383649 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:24.383616 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp" event={"ID":"a9ee0794-5103-4670-ab89-6a5129a64e3c","Type":"ContainerDied","Data":"75c370d289b1fc0085ed575de0e16f06f09e2295476b0a5a36e8e6308828d088"} Jul 16 21:40:24.383830 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:24.383673 2577 scope.go:117] "RemoveContainer" containerID="2987078abd95e0d6a9824f692866b1a481fa3168ed1bc97221283a79141ec6dc" Jul 16 21:40:24.383830 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:24.383627 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp" Jul 16 21:40:24.405966 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:24.405938 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp"] Jul 16 21:40:24.413253 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:24.413228 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-22pqp"] Jul 16 21:40:24.456672 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:24.456640 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9ee0794-5103-4670-ab89-6a5129a64e3c" path="/var/lib/kubelet/pods/a9ee0794-5103-4670-ab89-6a5129a64e3c/volumes" Jul 16 21:40:26.089867 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:26.089827 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll"] Jul 16 21:40:26.090373 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:26.090117 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" podUID="38a12dfe-2221-40d0-87e0-49f86741d888" containerName="istio-proxy" containerID="cri-o://4f934b225de0d0df20481699fd931a7ad38644694b5a843094276c2a8d2014c4" gracePeriod=30 Jul 16 21:40:31.343120 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.343091 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:40:31.421448 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.421415 2577 generic.go:356] "Generic (PLEG): container finished" podID="38a12dfe-2221-40d0-87e0-49f86741d888" containerID="4f934b225de0d0df20481699fd931a7ad38644694b5a843094276c2a8d2014c4" exitCode=0 Jul 16 21:40:31.421624 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.421485 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" Jul 16 21:40:31.421624 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.421501 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" event={"ID":"38a12dfe-2221-40d0-87e0-49f86741d888","Type":"ContainerDied","Data":"4f934b225de0d0df20481699fd931a7ad38644694b5a843094276c2a8d2014c4"} Jul 16 21:40:31.421624 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.421571 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll" event={"ID":"38a12dfe-2221-40d0-87e0-49f86741d888","Type":"ContainerDied","Data":"fefde41dc1f2102d59d6219d5ab32a8c8fea243f71c44ebaf03f130925498bcd"} Jul 16 21:40:31.421624 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.421606 2577 scope.go:117] "RemoveContainer" containerID="4f934b225de0d0df20481699fd931a7ad38644694b5a843094276c2a8d2014c4" Jul 16 21:40:31.430422 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.430381 2577 scope.go:117] "RemoveContainer" containerID="426c5e80050616af22eee916abaab44b92898aa6c1b277641d4b0af51a689f94" Jul 16 21:40:31.438620 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.438602 2577 scope.go:117] "RemoveContainer" containerID="4f934b225de0d0df20481699fd931a7ad38644694b5a843094276c2a8d2014c4" Jul 16 21:40:31.438903 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:40:31.438880 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f934b225de0d0df20481699fd931a7ad38644694b5a843094276c2a8d2014c4\": container with ID starting with 4f934b225de0d0df20481699fd931a7ad38644694b5a843094276c2a8d2014c4 not found: ID does not exist" containerID="4f934b225de0d0df20481699fd931a7ad38644694b5a843094276c2a8d2014c4" Jul 16 21:40:31.439013 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.438911 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f934b225de0d0df20481699fd931a7ad38644694b5a843094276c2a8d2014c4"} err="failed to get container status \"4f934b225de0d0df20481699fd931a7ad38644694b5a843094276c2a8d2014c4\": rpc error: code = NotFound desc = could not find container \"4f934b225de0d0df20481699fd931a7ad38644694b5a843094276c2a8d2014c4\": container with ID starting with 4f934b225de0d0df20481699fd931a7ad38644694b5a843094276c2a8d2014c4 not found: ID does not exist" Jul 16 21:40:31.439013 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.438931 2577 scope.go:117] "RemoveContainer" containerID="426c5e80050616af22eee916abaab44b92898aa6c1b277641d4b0af51a689f94" Jul 16 21:40:31.439220 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:40:31.439176 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"426c5e80050616af22eee916abaab44b92898aa6c1b277641d4b0af51a689f94\": container with ID starting with 426c5e80050616af22eee916abaab44b92898aa6c1b277641d4b0af51a689f94 not found: ID does not exist" containerID="426c5e80050616af22eee916abaab44b92898aa6c1b277641d4b0af51a689f94" Jul 16 21:40:31.439267 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.439229 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"426c5e80050616af22eee916abaab44b92898aa6c1b277641d4b0af51a689f94"} err="failed to get container status \"426c5e80050616af22eee916abaab44b92898aa6c1b277641d4b0af51a689f94\": rpc error: code = NotFound desc = could not find container \"426c5e80050616af22eee916abaab44b92898aa6c1b277641d4b0af51a689f94\": container with ID starting with 426c5e80050616af22eee916abaab44b92898aa6c1b277641d4b0af51a689f94 not found: ID does not exist" Jul 16 21:40:31.477092 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.477059 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a12dfe-2221-40d0-87e0-49f86741d888-istiod-ca-cert\") pod \"38a12dfe-2221-40d0-87e0-49f86741d888\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " Jul 16 21:40:31.477290 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.477132 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-credential-socket\") pod \"38a12dfe-2221-40d0-87e0-49f86741d888\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " Jul 16 21:40:31.477290 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.477153 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a12dfe-2221-40d0-87e0-49f86741d888-istio-podinfo\") pod \"38a12dfe-2221-40d0-87e0-49f86741d888\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " Jul 16 21:40:31.477290 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.477207 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7f8x\" (UniqueName: \"kubernetes.io/projected/38a12dfe-2221-40d0-87e0-49f86741d888-kube-api-access-j7f8x\") pod \"38a12dfe-2221-40d0-87e0-49f86741d888\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " Jul 16 21:40:31.477290 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.477243 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-workload-certs\") pod \"38a12dfe-2221-40d0-87e0-49f86741d888\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " Jul 16 21:40:31.477290 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.477290 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a12dfe-2221-40d0-87e0-49f86741d888-istio-token\") pod \"38a12dfe-2221-40d0-87e0-49f86741d888\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " Jul 16 21:40:31.477527 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.477336 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-workload-socket\") pod \"38a12dfe-2221-40d0-87e0-49f86741d888\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " Jul 16 21:40:31.477527 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.477364 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-istio-data\") pod \"38a12dfe-2221-40d0-87e0-49f86741d888\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " Jul 16 21:40:31.477527 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.477417 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-istio-envoy\") pod \"38a12dfe-2221-40d0-87e0-49f86741d888\" (UID: \"38a12dfe-2221-40d0-87e0-49f86741d888\") " Jul 16 21:40:31.477675 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.477514 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "38a12dfe-2221-40d0-87e0-49f86741d888" (UID: "38a12dfe-2221-40d0-87e0-49f86741d888"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:40:31.477675 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.477534 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a12dfe-2221-40d0-87e0-49f86741d888-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "38a12dfe-2221-40d0-87e0-49f86741d888" (UID: "38a12dfe-2221-40d0-87e0-49f86741d888"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:40:31.477675 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.477613 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "38a12dfe-2221-40d0-87e0-49f86741d888" (UID: "38a12dfe-2221-40d0-87e0-49f86741d888"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:40:31.477827 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.477765 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "38a12dfe-2221-40d0-87e0-49f86741d888" (UID: "38a12dfe-2221-40d0-87e0-49f86741d888"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:40:31.477888 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.477833 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a12dfe-2221-40d0-87e0-49f86741d888-istiod-ca-cert\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:40:31.477888 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.477853 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-credential-socket\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:40:31.477888 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.477868 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-workload-socket\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:40:31.478377 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.478353 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-istio-data" (OuterVolumeSpecName: "istio-data") pod "38a12dfe-2221-40d0-87e0-49f86741d888" (UID: "38a12dfe-2221-40d0-87e0-49f86741d888"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:40:31.479633 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.479601 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38a12dfe-2221-40d0-87e0-49f86741d888-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "38a12dfe-2221-40d0-87e0-49f86741d888" (UID: "38a12dfe-2221-40d0-87e0-49f86741d888"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 21:40:31.479744 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.479612 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a12dfe-2221-40d0-87e0-49f86741d888-kube-api-access-j7f8x" (OuterVolumeSpecName: "kube-api-access-j7f8x") pod "38a12dfe-2221-40d0-87e0-49f86741d888" (UID: "38a12dfe-2221-40d0-87e0-49f86741d888"). InnerVolumeSpecName "kube-api-access-j7f8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:40:31.479867 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.479846 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "38a12dfe-2221-40d0-87e0-49f86741d888" (UID: "38a12dfe-2221-40d0-87e0-49f86741d888"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:40:31.479958 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.479910 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a12dfe-2221-40d0-87e0-49f86741d888-istio-token" (OuterVolumeSpecName: "istio-token") pod "38a12dfe-2221-40d0-87e0-49f86741d888" (UID: "38a12dfe-2221-40d0-87e0-49f86741d888"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:40:31.578781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.578730 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-istio-envoy\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:40:31.578781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.578770 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a12dfe-2221-40d0-87e0-49f86741d888-istio-podinfo\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:40:31.578781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.578782 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7f8x\" (UniqueName: \"kubernetes.io/projected/38a12dfe-2221-40d0-87e0-49f86741d888-kube-api-access-j7f8x\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:40:31.579052 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.578794 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-workload-certs\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:40:31.579052 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.578807 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a12dfe-2221-40d0-87e0-49f86741d888-istio-token\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:40:31.579052 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.578819 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a12dfe-2221-40d0-87e0-49f86741d888-istio-data\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:40:31.744202 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.744161 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll"] Jul 16 21:40:31.748374 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:31.748341 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fc2328dc-openshift-default-594d7ff657-79pll"] Jul 16 21:40:32.450486 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:32.450452 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38a12dfe-2221-40d0-87e0-49f86741d888" path="/var/lib/kubelet/pods/38a12dfe-2221-40d0-87e0-49f86741d888/volumes" Jul 16 21:40:40.528311 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.528274 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85"] Jul 16 21:40:40.528798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.528781 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9ee0794-5103-4670-ab89-6a5129a64e3c" containerName="revoke-keys" Jul 16 21:40:40.528843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.528801 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9ee0794-5103-4670-ab89-6a5129a64e3c" containerName="revoke-keys" Jul 16 21:40:40.528843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.528815 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a12dfe-2221-40d0-87e0-49f86741d888" containerName="istio-proxy" Jul 16 21:40:40.528843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.528824 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a12dfe-2221-40d0-87e0-49f86741d888" containerName="istio-proxy" Jul 16 21:40:40.528843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.528839 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a12dfe-2221-40d0-87e0-49f86741d888" containerName="istio-proxy" Jul 16 21:40:40.528963 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.528848 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a12dfe-2221-40d0-87e0-49f86741d888" containerName="istio-proxy" Jul 16 21:40:40.528963 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.528860 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9ee0794-5103-4670-ab89-6a5129a64e3c" containerName="revoke-keys" Jul 16 21:40:40.528963 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.528870 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9ee0794-5103-4670-ab89-6a5129a64e3c" containerName="revoke-keys" Jul 16 21:40:40.529059 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.528974 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9ee0794-5103-4670-ab89-6a5129a64e3c" containerName="revoke-keys" Jul 16 21:40:40.529059 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.528987 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="17c0adb5-3334-4ce0-a333-98f7b5f06837" containerName="revoke-keys" Jul 16 21:40:40.529059 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.529000 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9ee0794-5103-4670-ab89-6a5129a64e3c" containerName="revoke-keys" Jul 16 21:40:40.529059 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.529009 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="38a12dfe-2221-40d0-87e0-49f86741d888" containerName="istio-proxy" Jul 16 21:40:40.529059 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.529018 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="38a12dfe-2221-40d0-87e0-49f86741d888" containerName="istio-proxy" Jul 16 21:40:40.533881 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.533858 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:40:40.536479 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.536458 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:40:40.537622 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.537602 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-0f8cd113-serving-cert\"" Jul 16 21:40:40.537622 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.537613 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xcg5c\"" Jul 16 21:40:40.537757 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.537609 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 21:40:40.541230 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.541207 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85"] Jul 16 21:40:40.659242 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.659177 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftpfj\" (UniqueName: \"kubernetes.io/projected/5f49f9ba-1661-404f-ac16-027e4375da41-kube-api-access-ftpfj\") pod \"maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85\" (UID: \"5f49f9ba-1661-404f-ac16-027e4375da41\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:40:40.659429 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.659307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f49f9ba-1661-404f-ac16-027e4375da41-maas-api-tls\") pod \"maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85\" (UID: \"5f49f9ba-1661-404f-ac16-027e4375da41\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:40:40.659429 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.659353 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f49f9ba-1661-404f-ac16-027e4375da41-openshift-service-ca\") pod \"maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85\" (UID: \"5f49f9ba-1661-404f-ac16-027e4375da41\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:40:40.760117 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.760078 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f49f9ba-1661-404f-ac16-027e4375da41-openshift-service-ca\") pod \"maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85\" (UID: \"5f49f9ba-1661-404f-ac16-027e4375da41\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:40:40.760352 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.760143 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftpfj\" (UniqueName: \"kubernetes.io/projected/5f49f9ba-1661-404f-ac16-027e4375da41-kube-api-access-ftpfj\") pod \"maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85\" (UID: \"5f49f9ba-1661-404f-ac16-027e4375da41\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:40:40.760352 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.760216 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f49f9ba-1661-404f-ac16-027e4375da41-maas-api-tls\") pod \"maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85\" (UID: \"5f49f9ba-1661-404f-ac16-027e4375da41\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:40:40.760352 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:40:40.760324 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-serving-cert: secret "maas-api-e2e-migrate-0f8cd113-serving-cert" not found Jul 16 21:40:40.760531 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:40:40.760402 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f49f9ba-1661-404f-ac16-027e4375da41-maas-api-tls podName:5f49f9ba-1661-404f-ac16-027e4375da41 nodeName:}" failed. No retries permitted until 2026-07-16 21:40:41.260381544 +0000 UTC m=+1857.435489715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f49f9ba-1661-404f-ac16-027e4375da41-maas-api-tls") pod "maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" (UID: "5f49f9ba-1661-404f-ac16-027e4375da41") : secret "maas-api-e2e-migrate-0f8cd113-serving-cert" not found Jul 16 21:40:40.760860 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.760834 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f49f9ba-1661-404f-ac16-027e4375da41-openshift-service-ca\") pod \"maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85\" (UID: \"5f49f9ba-1661-404f-ac16-027e4375da41\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:40:40.769456 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:40.769432 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftpfj\" (UniqueName: \"kubernetes.io/projected/5f49f9ba-1661-404f-ac16-027e4375da41-kube-api-access-ftpfj\") pod \"maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85\" (UID: \"5f49f9ba-1661-404f-ac16-027e4375da41\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:40:41.263874 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:41.263836 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f49f9ba-1661-404f-ac16-027e4375da41-maas-api-tls\") pod \"maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85\" (UID: \"5f49f9ba-1661-404f-ac16-027e4375da41\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:40:41.266393 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:41.266367 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f49f9ba-1661-404f-ac16-027e4375da41-maas-api-tls\") pod \"maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85\" (UID: \"5f49f9ba-1661-404f-ac16-027e4375da41\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:40:41.445686 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:41.445647 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:40:41.582849 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:41.582824 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85"] Jul 16 21:40:41.584546 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:40:41.584516 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f49f9ba_1661_404f_ac16_027e4375da41.slice/crio-1cc5b7c2c3c448bf8163949a2742857a7a8eccd3e0d135d19b8d66db36df713a WatchSource:0}: Error finding container 1cc5b7c2c3c448bf8163949a2742857a7a8eccd3e0d135d19b8d66db36df713a: Status 404 returned error can't find the container with id 1cc5b7c2c3c448bf8163949a2742857a7a8eccd3e0d135d19b8d66db36df713a Jul 16 21:40:42.479145 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:42.479106 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" event={"ID":"5f49f9ba-1661-404f-ac16-027e4375da41","Type":"ContainerStarted","Data":"1cc5b7c2c3c448bf8163949a2742857a7a8eccd3e0d135d19b8d66db36df713a"} Jul 16 21:40:44.493082 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:44.493038 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" event={"ID":"5f49f9ba-1661-404f-ac16-027e4375da41","Type":"ContainerStarted","Data":"894964e6713dde5cdf168b7d03ff800e4aeadd267998bb5a608f763a78039771"} Jul 16 21:40:44.493492 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:44.493147 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:40:44.514340 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:44.514280 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" podStartSLOduration=1.790366428 podStartE2EDuration="4.514263668s" totalImagesPullingTime="2.72389724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:40:44.511434593 +0000 UTC m=+1860.686542809" watchObservedRunningTime="2026-07-16 21:40:44.514263668 +0000 UTC m=+1860.689371941" Jul 16 21:40:44.971153 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:44.971115 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n"] Jul 16 21:40:44.971493 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:44.971480 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9ee0794-5103-4670-ab89-6a5129a64e3c" containerName="revoke-keys" Jul 16 21:40:44.971539 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:44.971495 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9ee0794-5103-4670-ab89-6a5129a64e3c" containerName="revoke-keys" Jul 16 21:40:44.971578 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:44.971558 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9ee0794-5103-4670-ab89-6a5129a64e3c" containerName="revoke-keys" Jul 16 21:40:44.974479 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:44.974463 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n" Jul 16 21:40:44.977136 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:44.977113 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:40:44.983046 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:44.983012 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n"] Jul 16 21:40:45.101280 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:45.101239 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795c4cd1-8471-4d72-a468-844bc9a1eccb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n\" (UID: \"795c4cd1-8471-4d72-a468-844bc9a1eccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n" Jul 16 21:40:45.201855 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:45.201812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795c4cd1-8471-4d72-a468-844bc9a1eccb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n\" (UID: \"795c4cd1-8471-4d72-a468-844bc9a1eccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n" Jul 16 21:40:45.202507 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:45.202489 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795c4cd1-8471-4d72-a468-844bc9a1eccb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n\" (UID: \"795c4cd1-8471-4d72-a468-844bc9a1eccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n" Jul 16 21:40:45.286543 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:45.286430 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n" Jul 16 21:40:45.435201 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:45.435160 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n"] Jul 16 21:40:45.499466 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:45.499426 2577 generic.go:356] "Generic (PLEG): container finished" podID="5f49f9ba-1661-404f-ac16-027e4375da41" containerID="894964e6713dde5cdf168b7d03ff800e4aeadd267998bb5a608f763a78039771" exitCode=1 Jul 16 21:40:45.499927 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:45.499509 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" event={"ID":"5f49f9ba-1661-404f-ac16-027e4375da41","Type":"ContainerDied","Data":"894964e6713dde5cdf168b7d03ff800e4aeadd267998bb5a608f763a78039771"} Jul 16 21:40:45.499927 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:45.499823 2577 scope.go:117] "RemoveContainer" containerID="894964e6713dde5cdf168b7d03ff800e4aeadd267998bb5a608f763a78039771" Jul 16 21:40:45.500929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:45.500910 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n" event={"ID":"795c4cd1-8471-4d72-a468-844bc9a1eccb","Type":"ContainerStarted","Data":"ed5bdc78b06ab3c7e3f53044d33bee91168988da4ca60e0af9e1fd39bfbf5b01"} Jul 16 21:40:46.507891 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:46.507796 2577 generic.go:356] "Generic (PLEG): container finished" podID="795c4cd1-8471-4d72-a468-844bc9a1eccb" containerID="eb59293037ea7e45ab30081f1cdeed7753b47c870563b6202279d6732402948c" exitCode=7 Jul 16 21:40:46.508325 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:46.507891 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n" event={"ID":"795c4cd1-8471-4d72-a468-844bc9a1eccb","Type":"ContainerDied","Data":"eb59293037ea7e45ab30081f1cdeed7753b47c870563b6202279d6732402948c"} Jul 16 21:40:46.508325 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:46.508044 2577 scope.go:117] "RemoveContainer" containerID="eb59293037ea7e45ab30081f1cdeed7753b47c870563b6202279d6732402948c" Jul 16 21:40:46.509882 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:46.509825 2577 generic.go:356] "Generic (PLEG): container finished" podID="5f49f9ba-1661-404f-ac16-027e4375da41" containerID="e5f47ed9424b55f33d1a9487f01d8f71c1d998d17d4919667466b2062dc23730" exitCode=1 Jul 16 21:40:46.509882 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:46.509864 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" event={"ID":"5f49f9ba-1661-404f-ac16-027e4375da41","Type":"ContainerDied","Data":"e5f47ed9424b55f33d1a9487f01d8f71c1d998d17d4919667466b2062dc23730"} Jul 16 21:40:46.510056 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:46.509903 2577 scope.go:117] "RemoveContainer" containerID="894964e6713dde5cdf168b7d03ff800e4aeadd267998bb5a608f763a78039771" Jul 16 21:40:46.510175 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:46.510158 2577 scope.go:117] "RemoveContainer" containerID="e5f47ed9424b55f33d1a9487f01d8f71c1d998d17d4919667466b2062dc23730" Jul 16 21:40:46.510397 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:40:46.510377 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85_odh-ai-gateway-infra(5f49f9ba-1661-404f-ac16-027e4375da41)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" Jul 16 21:40:47.516847 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:47.516817 2577 scope.go:117] "RemoveContainer" containerID="e5f47ed9424b55f33d1a9487f01d8f71c1d998d17d4919667466b2062dc23730" Jul 16 21:40:47.517288 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:40:47.517048 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85_odh-ai-gateway-infra(5f49f9ba-1661-404f-ac16-027e4375da41)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" Jul 16 21:40:47.518309 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:47.518281 2577 generic.go:356] "Generic (PLEG): container finished" podID="795c4cd1-8471-4d72-a468-844bc9a1eccb" containerID="2ae6051726facc308bae5ab21428edf376a8c7287a73cb698497a5044f67c55b" exitCode=7 Jul 16 21:40:47.518397 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:47.518360 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n" event={"ID":"795c4cd1-8471-4d72-a468-844bc9a1eccb","Type":"ContainerDied","Data":"2ae6051726facc308bae5ab21428edf376a8c7287a73cb698497a5044f67c55b"} Jul 16 21:40:47.518447 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:47.518416 2577 scope.go:117] "RemoveContainer" containerID="eb59293037ea7e45ab30081f1cdeed7753b47c870563b6202279d6732402948c" Jul 16 21:40:47.518574 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:47.518561 2577 scope.go:117] "RemoveContainer" containerID="2ae6051726facc308bae5ab21428edf376a8c7287a73cb698497a5044f67c55b" Jul 16 21:40:47.518811 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:40:47.518791 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n_odh-ai-gateway-infra(795c4cd1-8471-4d72-a468-844bc9a1eccb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n" podUID="795c4cd1-8471-4d72-a468-844bc9a1eccb" Jul 16 21:40:48.524586 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:48.524557 2577 scope.go:117] "RemoveContainer" containerID="2ae6051726facc308bae5ab21428edf376a8c7287a73cb698497a5044f67c55b" Jul 16 21:40:48.524979 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:40:48.524774 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n_odh-ai-gateway-infra(795c4cd1-8471-4d72-a468-844bc9a1eccb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n" podUID="795c4cd1-8471-4d72-a468-844bc9a1eccb" Jul 16 21:40:49.493653 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:49.493619 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:40:49.494048 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:49.494030 2577 scope.go:117] "RemoveContainer" containerID="e5f47ed9424b55f33d1a9487f01d8f71c1d998d17d4919667466b2062dc23730" Jul 16 21:40:49.494304 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:40:49.494282 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85_odh-ai-gateway-infra(5f49f9ba-1661-404f-ac16-027e4375da41)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" Jul 16 21:40:51.446134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:51.446106 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:40:51.446627 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:51.446528 2577 scope.go:117] "RemoveContainer" containerID="e5f47ed9424b55f33d1a9487f01d8f71c1d998d17d4919667466b2062dc23730" Jul 16 21:40:51.446735 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:40:51.446714 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85_odh-ai-gateway-infra(5f49f9ba-1661-404f-ac16-027e4375da41)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" Jul 16 21:40:56.161813 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:56.161782 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg"] Jul 16 21:40:56.166323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:56.166301 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg" Jul 16 21:40:56.179890 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:56.179854 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg"] Jul 16 21:40:56.198384 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:56.198348 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd9f1a14-ec8a-4b39-8570-2cd667bc2992-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg\" (UID: \"fd9f1a14-ec8a-4b39-8570-2cd667bc2992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg" Jul 16 21:40:56.298901 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:56.298868 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd9f1a14-ec8a-4b39-8570-2cd667bc2992-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg\" (UID: \"fd9f1a14-ec8a-4b39-8570-2cd667bc2992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg" Jul 16 21:40:56.299509 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:56.299489 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd9f1a14-ec8a-4b39-8570-2cd667bc2992-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg\" (UID: \"fd9f1a14-ec8a-4b39-8570-2cd667bc2992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg" Jul 16 21:40:56.487204 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:56.487075 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg" Jul 16 21:40:56.618867 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:56.618842 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg"] Jul 16 21:40:56.620533 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:40:56.620499 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9f1a14_ec8a_4b39_8570_2cd667bc2992.slice/crio-78acf1e2669ae4a48dcc16e9ba2165fe94d4539ed8b401356bab08056a385379 WatchSource:0}: Error finding container 78acf1e2669ae4a48dcc16e9ba2165fe94d4539ed8b401356bab08056a385379: Status 404 returned error can't find the container with id 78acf1e2669ae4a48dcc16e9ba2165fe94d4539ed8b401356bab08056a385379 Jul 16 21:40:57.572886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:57.572799 2577 generic.go:356] "Generic (PLEG): container finished" podID="fd9f1a14-ec8a-4b39-8570-2cd667bc2992" containerID="c86e56c24f6c4f543dea7e69fbcedf297c6a23ae0b8f1b7d42c67fc65126048b" exitCode=6 Jul 16 21:40:57.573263 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:57.572882 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg" event={"ID":"fd9f1a14-ec8a-4b39-8570-2cd667bc2992","Type":"ContainerDied","Data":"c86e56c24f6c4f543dea7e69fbcedf297c6a23ae0b8f1b7d42c67fc65126048b"} Jul 16 21:40:57.573263 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:57.572931 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg" event={"ID":"fd9f1a14-ec8a-4b39-8570-2cd667bc2992","Type":"ContainerStarted","Data":"78acf1e2669ae4a48dcc16e9ba2165fe94d4539ed8b401356bab08056a385379"} Jul 16 21:40:57.573263 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:57.573042 2577 scope.go:117] "RemoveContainer" containerID="c86e56c24f6c4f543dea7e69fbcedf297c6a23ae0b8f1b7d42c67fc65126048b" Jul 16 21:40:58.580161 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:58.580066 2577 generic.go:356] "Generic (PLEG): container finished" podID="fd9f1a14-ec8a-4b39-8570-2cd667bc2992" containerID="97c0431c3b6b8a64f50996a1a55fcbffb443fd09031b93ae56e67c69b4a349b3" exitCode=6 Jul 16 21:40:58.580161 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:58.580101 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg" event={"ID":"fd9f1a14-ec8a-4b39-8570-2cd667bc2992","Type":"ContainerDied","Data":"97c0431c3b6b8a64f50996a1a55fcbffb443fd09031b93ae56e67c69b4a349b3"} Jul 16 21:40:58.580161 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:58.580149 2577 scope.go:117] "RemoveContainer" containerID="c86e56c24f6c4f543dea7e69fbcedf297c6a23ae0b8f1b7d42c67fc65126048b" Jul 16 21:40:58.580766 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:58.580443 2577 scope.go:117] "RemoveContainer" containerID="97c0431c3b6b8a64f50996a1a55fcbffb443fd09031b93ae56e67c69b4a349b3" Jul 16 21:40:58.580766 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:40:58.580687 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg_odh-ai-gateway-infra(fd9f1a14-ec8a-4b39-8570-2cd667bc2992)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg" podUID="fd9f1a14-ec8a-4b39-8570-2cd667bc2992" Jul 16 21:40:59.586846 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:40:59.586818 2577 scope.go:117] "RemoveContainer" containerID="97c0431c3b6b8a64f50996a1a55fcbffb443fd09031b93ae56e67c69b4a349b3" Jul 16 21:40:59.587256 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:40:59.587016 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg_odh-ai-gateway-infra(fd9f1a14-ec8a-4b39-8570-2cd667bc2992)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg" podUID="fd9f1a14-ec8a-4b39-8570-2cd667bc2992" Jul 16 21:41:01.445165 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:01.445135 2577 scope.go:117] "RemoveContainer" containerID="2ae6051726facc308bae5ab21428edf376a8c7287a73cb698497a5044f67c55b" Jul 16 21:41:02.602258 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:02.602226 2577 generic.go:356] "Generic (PLEG): container finished" podID="795c4cd1-8471-4d72-a468-844bc9a1eccb" containerID="b032fb77c4286969cb07959e1a6cbfbc90fa9c21c601597b6f29539f382c3ee2" exitCode=7 Jul 16 21:41:02.602626 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:02.602303 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n" event={"ID":"795c4cd1-8471-4d72-a468-844bc9a1eccb","Type":"ContainerDied","Data":"b032fb77c4286969cb07959e1a6cbfbc90fa9c21c601597b6f29539f382c3ee2"} Jul 16 21:41:02.602626 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:02.602369 2577 scope.go:117] "RemoveContainer" containerID="2ae6051726facc308bae5ab21428edf376a8c7287a73cb698497a5044f67c55b" Jul 16 21:41:02.602626 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:02.602588 2577 scope.go:117] "RemoveContainer" containerID="b032fb77c4286969cb07959e1a6cbfbc90fa9c21c601597b6f29539f382c3ee2" Jul 16 21:41:02.602840 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:41:02.602822 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n_odh-ai-gateway-infra(795c4cd1-8471-4d72-a468-844bc9a1eccb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n" podUID="795c4cd1-8471-4d72-a468-844bc9a1eccb" Jul 16 21:41:03.445493 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:03.445463 2577 scope.go:117] "RemoveContainer" containerID="e5f47ed9424b55f33d1a9487f01d8f71c1d998d17d4919667466b2062dc23730" Jul 16 21:41:03.628092 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:03.628053 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n"] Jul 16 21:41:03.763314 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:03.763288 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n" Jul 16 21:41:03.848927 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:03.848889 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795c4cd1-8471-4d72-a468-844bc9a1eccb-maas-api-service-ca\") pod \"795c4cd1-8471-4d72-a468-844bc9a1eccb\" (UID: \"795c4cd1-8471-4d72-a468-844bc9a1eccb\") " Jul 16 21:41:03.849307 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:03.849286 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795c4cd1-8471-4d72-a468-844bc9a1eccb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "795c4cd1-8471-4d72-a468-844bc9a1eccb" (UID: "795c4cd1-8471-4d72-a468-844bc9a1eccb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:03.950068 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:03.950027 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795c4cd1-8471-4d72-a468-844bc9a1eccb-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:41:04.615982 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:04.615947 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n" event={"ID":"795c4cd1-8471-4d72-a468-844bc9a1eccb","Type":"ContainerDied","Data":"ed5bdc78b06ab3c7e3f53044d33bee91168988da4ca60e0af9e1fd39bfbf5b01"} Jul 16 21:41:04.616177 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:04.615989 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n" Jul 16 21:41:04.616177 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:04.616012 2577 scope.go:117] "RemoveContainer" containerID="b032fb77c4286969cb07959e1a6cbfbc90fa9c21c601597b6f29539f382c3ee2" Jul 16 21:41:04.633455 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:04.633428 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n"] Jul 16 21:41:04.639032 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:04.639008 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-2w96n"] Jul 16 21:41:05.624004 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:05.623971 2577 generic.go:356] "Generic (PLEG): container finished" podID="5f49f9ba-1661-404f-ac16-027e4375da41" containerID="1b374d464ff20e685b3b53968086e54a26c51aabd86ac0a752edd1240b1e7680" exitCode=1 Jul 16 21:41:05.624224 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:05.624044 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" event={"ID":"5f49f9ba-1661-404f-ac16-027e4375da41","Type":"ContainerDied","Data":"1b374d464ff20e685b3b53968086e54a26c51aabd86ac0a752edd1240b1e7680"} Jul 16 21:41:05.624224 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:05.624096 2577 scope.go:117] "RemoveContainer" containerID="e5f47ed9424b55f33d1a9487f01d8f71c1d998d17d4919667466b2062dc23730" Jul 16 21:41:05.624485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:05.624468 2577 scope.go:117] "RemoveContainer" containerID="1b374d464ff20e685b3b53968086e54a26c51aabd86ac0a752edd1240b1e7680" Jul 16 21:41:05.624696 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:41:05.624673 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85_odh-ai-gateway-infra(5f49f9ba-1661-404f-ac16-027e4375da41)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" Jul 16 21:41:06.451054 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:06.451023 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="795c4cd1-8471-4d72-a468-844bc9a1eccb" path="/var/lib/kubelet/pods/795c4cd1-8471-4d72-a468-844bc9a1eccb/volumes" Jul 16 21:41:09.494249 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:09.494217 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:41:09.494650 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:09.494607 2577 scope.go:117] "RemoveContainer" containerID="1b374d464ff20e685b3b53968086e54a26c51aabd86ac0a752edd1240b1e7680" Jul 16 21:41:09.494798 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:41:09.494778 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85_odh-ai-gateway-infra(5f49f9ba-1661-404f-ac16-027e4375da41)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" Jul 16 21:41:11.445886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:11.445848 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:41:11.446515 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:11.446370 2577 scope.go:117] "RemoveContainer" containerID="1b374d464ff20e685b3b53968086e54a26c51aabd86ac0a752edd1240b1e7680" Jul 16 21:41:11.446627 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:41:11.446603 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85_odh-ai-gateway-infra(5f49f9ba-1661-404f-ac16-027e4375da41)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" Jul 16 21:41:13.444994 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:13.444963 2577 scope.go:117] "RemoveContainer" containerID="97c0431c3b6b8a64f50996a1a55fcbffb443fd09031b93ae56e67c69b4a349b3" Jul 16 21:41:14.671300 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:14.671264 2577 generic.go:356] "Generic (PLEG): container finished" podID="fd9f1a14-ec8a-4b39-8570-2cd667bc2992" containerID="7019f2b72eaafd2b4c48e37a73ebd6876442cebbd967ef5c6ddeb50811d979da" exitCode=6 Jul 16 21:41:14.671724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:14.671321 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg" event={"ID":"fd9f1a14-ec8a-4b39-8570-2cd667bc2992","Type":"ContainerDied","Data":"7019f2b72eaafd2b4c48e37a73ebd6876442cebbd967ef5c6ddeb50811d979da"} Jul 16 21:41:14.671724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:14.671371 2577 scope.go:117] "RemoveContainer" containerID="97c0431c3b6b8a64f50996a1a55fcbffb443fd09031b93ae56e67c69b4a349b3" Jul 16 21:41:14.671724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:14.671681 2577 scope.go:117] "RemoveContainer" containerID="7019f2b72eaafd2b4c48e37a73ebd6876442cebbd967ef5c6ddeb50811d979da" Jul 16 21:41:14.671907 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:41:14.671881 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg_odh-ai-gateway-infra(fd9f1a14-ec8a-4b39-8570-2cd667bc2992)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg" podUID="fd9f1a14-ec8a-4b39-8570-2cd667bc2992" Jul 16 21:41:15.698824 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:15.698792 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg"] Jul 16 21:41:15.838729 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:15.838692 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg" Jul 16 21:41:15.956575 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:15.956481 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd9f1a14-ec8a-4b39-8570-2cd667bc2992-maas-api-service-ca\") pod \"fd9f1a14-ec8a-4b39-8570-2cd667bc2992\" (UID: \"fd9f1a14-ec8a-4b39-8570-2cd667bc2992\") " Jul 16 21:41:15.956813 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:15.956790 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9f1a14-ec8a-4b39-8570-2cd667bc2992-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fd9f1a14-ec8a-4b39-8570-2cd667bc2992" (UID: "fd9f1a14-ec8a-4b39-8570-2cd667bc2992"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:16.058098 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:16.058040 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd9f1a14-ec8a-4b39-8570-2cd667bc2992-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:41:16.684207 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:16.684158 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg" event={"ID":"fd9f1a14-ec8a-4b39-8570-2cd667bc2992","Type":"ContainerDied","Data":"78acf1e2669ae4a48dcc16e9ba2165fe94d4539ed8b401356bab08056a385379"} Jul 16 21:41:16.684207 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:16.684199 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg" Jul 16 21:41:16.684428 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:16.684242 2577 scope.go:117] "RemoveContainer" containerID="7019f2b72eaafd2b4c48e37a73ebd6876442cebbd967ef5c6ddeb50811d979da" Jul 16 21:41:16.701167 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:16.701143 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg"] Jul 16 21:41:16.708933 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:16.708877 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-g2cgg"] Jul 16 21:41:18.450852 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:18.450817 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd9f1a14-ec8a-4b39-8570-2cd667bc2992" path="/var/lib/kubelet/pods/fd9f1a14-ec8a-4b39-8570-2cd667bc2992/volumes" Jul 16 21:41:23.445251 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:23.445217 2577 scope.go:117] "RemoveContainer" containerID="1b374d464ff20e685b3b53968086e54a26c51aabd86ac0a752edd1240b1e7680" Jul 16 21:41:23.445685 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:41:23.445394 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85_odh-ai-gateway-infra(5f49f9ba-1661-404f-ac16-027e4375da41)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" Jul 16 21:41:35.026544 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.026505 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h"] Jul 16 21:41:35.026932 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.026875 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795c4cd1-8471-4d72-a468-844bc9a1eccb" containerName="revoke-keys" Jul 16 21:41:35.026932 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.026886 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="795c4cd1-8471-4d72-a468-844bc9a1eccb" containerName="revoke-keys" Jul 16 21:41:35.026932 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.026901 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795c4cd1-8471-4d72-a468-844bc9a1eccb" containerName="revoke-keys" Jul 16 21:41:35.026932 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.026907 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="795c4cd1-8471-4d72-a468-844bc9a1eccb" containerName="revoke-keys" Jul 16 21:41:35.026932 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.026912 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd9f1a14-ec8a-4b39-8570-2cd667bc2992" containerName="revoke-keys" Jul 16 21:41:35.026932 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.026917 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd9f1a14-ec8a-4b39-8570-2cd667bc2992" containerName="revoke-keys" Jul 16 21:41:35.026932 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.026923 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd9f1a14-ec8a-4b39-8570-2cd667bc2992" containerName="revoke-keys" Jul 16 21:41:35.026932 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.026928 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd9f1a14-ec8a-4b39-8570-2cd667bc2992" containerName="revoke-keys" Jul 16 21:41:35.027250 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.027001 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="795c4cd1-8471-4d72-a468-844bc9a1eccb" containerName="revoke-keys" Jul 16 21:41:35.027250 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.027010 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd9f1a14-ec8a-4b39-8570-2cd667bc2992" containerName="revoke-keys" Jul 16 21:41:35.027250 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.027017 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="795c4cd1-8471-4d72-a468-844bc9a1eccb" containerName="revoke-keys" Jul 16 21:41:35.027250 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.027022 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd9f1a14-ec8a-4b39-8570-2cd667bc2992" containerName="revoke-keys" Jul 16 21:41:35.027250 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.027029 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd9f1a14-ec8a-4b39-8570-2cd667bc2992" containerName="revoke-keys" Jul 16 21:41:35.029266 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.029250 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" Jul 16 21:41:35.032208 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.032175 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:41:35.040444 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.040418 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h"] Jul 16 21:41:35.139497 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.139464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8151dd96-c41c-43ad-9575-28f3275a3e7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h\" (UID: \"8151dd96-c41c-43ad-9575-28f3275a3e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" Jul 16 21:41:35.240930 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.240889 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8151dd96-c41c-43ad-9575-28f3275a3e7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h\" (UID: \"8151dd96-c41c-43ad-9575-28f3275a3e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" Jul 16 21:41:35.241597 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.241571 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8151dd96-c41c-43ad-9575-28f3275a3e7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h\" (UID: \"8151dd96-c41c-43ad-9575-28f3275a3e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" Jul 16 21:41:35.340443 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.340355 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" Jul 16 21:41:35.479154 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.479121 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h"] Jul 16 21:41:35.483006 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:41:35.482964 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8151dd96_c41c_43ad_9575_28f3275a3e7f.slice/crio-bad8a51dc8f3788a5737101f7312572636107acd23f4748bdedde1404e38e2e0 WatchSource:0}: Error finding container bad8a51dc8f3788a5737101f7312572636107acd23f4748bdedde1404e38e2e0: Status 404 returned error can't find the container with id bad8a51dc8f3788a5737101f7312572636107acd23f4748bdedde1404e38e2e0 Jul 16 21:41:35.789477 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.789437 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" event={"ID":"8151dd96-c41c-43ad-9575-28f3275a3e7f","Type":"ContainerStarted","Data":"6594386e53c0560351473c974b2da1fbff37922e50847e0f7f8e5e1b69ac618d"} Jul 16 21:41:35.789656 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.789502 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" event={"ID":"8151dd96-c41c-43ad-9575-28f3275a3e7f","Type":"ContainerStarted","Data":"bad8a51dc8f3788a5737101f7312572636107acd23f4748bdedde1404e38e2e0"} Jul 16 21:41:35.806957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:35.806901 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" podStartSLOduration=0.806882909 podStartE2EDuration="806.882909ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:41:35.805811003 +0000 UTC m=+1911.980919197" watchObservedRunningTime="2026-07-16 21:41:35.806882909 +0000 UTC m=+1911.981991101" Jul 16 21:41:36.796048 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:36.796013 2577 generic.go:356] "Generic (PLEG): container finished" podID="8151dd96-c41c-43ad-9575-28f3275a3e7f" containerID="6594386e53c0560351473c974b2da1fbff37922e50847e0f7f8e5e1b69ac618d" exitCode=7 Jul 16 21:41:36.796488 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:36.796101 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" event={"ID":"8151dd96-c41c-43ad-9575-28f3275a3e7f","Type":"ContainerDied","Data":"6594386e53c0560351473c974b2da1fbff37922e50847e0f7f8e5e1b69ac618d"} Jul 16 21:41:36.796488 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:36.796265 2577 scope.go:117] "RemoveContainer" containerID="6594386e53c0560351473c974b2da1fbff37922e50847e0f7f8e5e1b69ac618d" Jul 16 21:41:37.445316 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:37.445287 2577 scope.go:117] "RemoveContainer" containerID="1b374d464ff20e685b3b53968086e54a26c51aabd86ac0a752edd1240b1e7680" Jul 16 21:41:37.802715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:37.802625 2577 generic.go:356] "Generic (PLEG): container finished" podID="8151dd96-c41c-43ad-9575-28f3275a3e7f" containerID="48cf41d545ece2f5c52b42a237ccb092bfb5e00f97a7702d747ec93c3f59a7dc" exitCode=7 Jul 16 21:41:37.802715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:37.802703 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" event={"ID":"8151dd96-c41c-43ad-9575-28f3275a3e7f","Type":"ContainerDied","Data":"48cf41d545ece2f5c52b42a237ccb092bfb5e00f97a7702d747ec93c3f59a7dc"} Jul 16 21:41:37.803283 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:37.802756 2577 scope.go:117] "RemoveContainer" containerID="6594386e53c0560351473c974b2da1fbff37922e50847e0f7f8e5e1b69ac618d" Jul 16 21:41:37.803283 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:37.802995 2577 scope.go:117] "RemoveContainer" containerID="48cf41d545ece2f5c52b42a237ccb092bfb5e00f97a7702d747ec93c3f59a7dc" Jul 16 21:41:37.803283 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:41:37.803264 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h_odh-ai-gateway-infra(8151dd96-c41c-43ad-9575-28f3275a3e7f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" podUID="8151dd96-c41c-43ad-9575-28f3275a3e7f" Jul 16 21:41:38.809717 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:38.809626 2577 generic.go:356] "Generic (PLEG): container finished" podID="5f49f9ba-1661-404f-ac16-027e4375da41" containerID="552627bcdf419e08bd08c52c2dc877073e56816eb16d3b2dd45f9ae561d95be8" exitCode=1 Jul 16 21:41:38.809717 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:38.809704 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" event={"ID":"5f49f9ba-1661-404f-ac16-027e4375da41","Type":"ContainerDied","Data":"552627bcdf419e08bd08c52c2dc877073e56816eb16d3b2dd45f9ae561d95be8"} Jul 16 21:41:38.810258 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:38.809772 2577 scope.go:117] "RemoveContainer" containerID="1b374d464ff20e685b3b53968086e54a26c51aabd86ac0a752edd1240b1e7680" Jul 16 21:41:38.810258 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:38.810110 2577 scope.go:117] "RemoveContainer" containerID="552627bcdf419e08bd08c52c2dc877073e56816eb16d3b2dd45f9ae561d95be8" Jul 16 21:41:38.810384 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:41:38.810369 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85_odh-ai-gateway-infra(5f49f9ba-1661-404f-ac16-027e4375da41)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" Jul 16 21:41:38.811485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:38.811470 2577 scope.go:117] "RemoveContainer" containerID="48cf41d545ece2f5c52b42a237ccb092bfb5e00f97a7702d747ec93c3f59a7dc" Jul 16 21:41:38.811657 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:41:38.811640 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h_odh-ai-gateway-infra(8151dd96-c41c-43ad-9575-28f3275a3e7f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" podUID="8151dd96-c41c-43ad-9575-28f3275a3e7f" Jul 16 21:41:39.493922 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:39.493879 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:41:39.818556 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:39.818469 2577 scope.go:117] "RemoveContainer" containerID="552627bcdf419e08bd08c52c2dc877073e56816eb16d3b2dd45f9ae561d95be8" Jul 16 21:41:39.819014 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:41:39.818729 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85_odh-ai-gateway-infra(5f49f9ba-1661-404f-ac16-027e4375da41)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" Jul 16 21:41:41.446782 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:41.446736 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:41:41.447235 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:41.447217 2577 scope.go:117] "RemoveContainer" containerID="552627bcdf419e08bd08c52c2dc877073e56816eb16d3b2dd45f9ae561d95be8" Jul 16 21:41:41.447428 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:41:41.447406 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85_odh-ai-gateway-infra(5f49f9ba-1661-404f-ac16-027e4375da41)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" Jul 16 21:41:50.445202 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:50.445155 2577 scope.go:117] "RemoveContainer" containerID="48cf41d545ece2f5c52b42a237ccb092bfb5e00f97a7702d747ec93c3f59a7dc" Jul 16 21:41:50.627865 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:50.627831 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85"] Jul 16 21:41:50.763160 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:50.763133 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:41:50.880782 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:50.880754 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" Jul 16 21:41:50.880983 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:50.880760 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85" event={"ID":"5f49f9ba-1661-404f-ac16-027e4375da41","Type":"ContainerDied","Data":"1cc5b7c2c3c448bf8163949a2742857a7a8eccd3e0d135d19b8d66db36df713a"} Jul 16 21:41:50.880983 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:50.880905 2577 scope.go:117] "RemoveContainer" containerID="552627bcdf419e08bd08c52c2dc877073e56816eb16d3b2dd45f9ae561d95be8" Jul 16 21:41:50.882485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:50.882462 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftpfj\" (UniqueName: \"kubernetes.io/projected/5f49f9ba-1661-404f-ac16-027e4375da41-kube-api-access-ftpfj\") pod \"5f49f9ba-1661-404f-ac16-027e4375da41\" (UID: \"5f49f9ba-1661-404f-ac16-027e4375da41\") " Jul 16 21:41:50.882591 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:50.882511 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f49f9ba-1661-404f-ac16-027e4375da41-maas-api-tls\") pod \"5f49f9ba-1661-404f-ac16-027e4375da41\" (UID: \"5f49f9ba-1661-404f-ac16-027e4375da41\") " Jul 16 21:41:50.882591 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:50.882555 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f49f9ba-1661-404f-ac16-027e4375da41-openshift-service-ca\") pod \"5f49f9ba-1661-404f-ac16-027e4375da41\" (UID: \"5f49f9ba-1661-404f-ac16-027e4375da41\") " Jul 16 21:41:50.882885 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:50.882810 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" event={"ID":"8151dd96-c41c-43ad-9575-28f3275a3e7f","Type":"ContainerStarted","Data":"81e2cf815b3823513b9463b59c0ca7e65fe937f7553eb7d03d43a11fd4c25e8e"} Jul 16 21:41:50.883060 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:50.883004 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f49f9ba-1661-404f-ac16-027e4375da41-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5f49f9ba-1661-404f-ac16-027e4375da41" (UID: "5f49f9ba-1661-404f-ac16-027e4375da41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:50.884829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:50.884802 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f49f9ba-1661-404f-ac16-027e4375da41-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5f49f9ba-1661-404f-ac16-027e4375da41" (UID: "5f49f9ba-1661-404f-ac16-027e4375da41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:41:50.885073 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:50.885039 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f49f9ba-1661-404f-ac16-027e4375da41-kube-api-access-ftpfj" (OuterVolumeSpecName: "kube-api-access-ftpfj") pod "5f49f9ba-1661-404f-ac16-027e4375da41" (UID: "5f49f9ba-1661-404f-ac16-027e4375da41"). InnerVolumeSpecName "kube-api-access-ftpfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:41:50.984076 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:50.983987 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftpfj\" (UniqueName: \"kubernetes.io/projected/5f49f9ba-1661-404f-ac16-027e4375da41-kube-api-access-ftpfj\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.984076 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:50.984022 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f49f9ba-1661-404f-ac16-027e4375da41-maas-api-tls\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:41:50.984076 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:50.984035 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f49f9ba-1661-404f-ac16-027e4375da41-openshift-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:41:51.247666 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:51.247577 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85"] Jul 16 21:41:51.262197 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:51.262156 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0f8cd113-657fcb6bbb-4pj85"] Jul 16 21:41:51.890840 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:51.890809 2577 generic.go:356] "Generic (PLEG): container finished" podID="8151dd96-c41c-43ad-9575-28f3275a3e7f" containerID="81e2cf815b3823513b9463b59c0ca7e65fe937f7553eb7d03d43a11fd4c25e8e" exitCode=6 Jul 16 21:41:51.891292 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:51.890892 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" event={"ID":"8151dd96-c41c-43ad-9575-28f3275a3e7f","Type":"ContainerDied","Data":"81e2cf815b3823513b9463b59c0ca7e65fe937f7553eb7d03d43a11fd4c25e8e"} Jul 16 21:41:51.891292 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:51.890987 2577 scope.go:117] "RemoveContainer" containerID="48cf41d545ece2f5c52b42a237ccb092bfb5e00f97a7702d747ec93c3f59a7dc" Jul 16 21:41:51.891292 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:51.891285 2577 scope.go:117] "RemoveContainer" containerID="81e2cf815b3823513b9463b59c0ca7e65fe937f7553eb7d03d43a11fd4c25e8e" Jul 16 21:41:51.891534 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:41:51.891514 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h_odh-ai-gateway-infra(8151dd96-c41c-43ad-9575-28f3275a3e7f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" podUID="8151dd96-c41c-43ad-9575-28f3275a3e7f" Jul 16 21:41:51.915738 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:51.915709 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h"] Jul 16 21:41:52.450959 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:52.450925 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" path="/var/lib/kubelet/pods/5f49f9ba-1661-404f-ac16-027e4375da41/volumes" Jul 16 21:41:53.030220 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:53.030178 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" Jul 16 21:41:53.097082 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:53.097041 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8151dd96-c41c-43ad-9575-28f3275a3e7f-maas-api-service-ca\") pod \"8151dd96-c41c-43ad-9575-28f3275a3e7f\" (UID: \"8151dd96-c41c-43ad-9575-28f3275a3e7f\") " Jul 16 21:41:53.097432 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:53.097408 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8151dd96-c41c-43ad-9575-28f3275a3e7f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8151dd96-c41c-43ad-9575-28f3275a3e7f" (UID: "8151dd96-c41c-43ad-9575-28f3275a3e7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:41:53.197975 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:53.197880 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8151dd96-c41c-43ad-9575-28f3275a3e7f-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:41:53.903568 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:53.903530 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" event={"ID":"8151dd96-c41c-43ad-9575-28f3275a3e7f","Type":"ContainerDied","Data":"bad8a51dc8f3788a5737101f7312572636107acd23f4748bdedde1404e38e2e0"} Jul 16 21:41:53.903815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:53.903597 2577 scope.go:117] "RemoveContainer" containerID="81e2cf815b3823513b9463b59c0ca7e65fe937f7553eb7d03d43a11fd4c25e8e" Jul 16 21:41:53.903815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:53.903545 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h" Jul 16 21:41:53.927639 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:53.927607 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h"] Jul 16 21:41:53.934550 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:53.934521 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lxt2h"] Jul 16 21:41:54.450673 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:54.450638 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8151dd96-c41c-43ad-9575-28f3275a3e7f" path="/var/lib/kubelet/pods/8151dd96-c41c-43ad-9575-28f3275a3e7f/volumes" Jul 16 21:41:56.214512 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214478 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm"] Jul 16 21:41:56.214929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214815 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" containerName="maas-api" Jul 16 21:41:56.214929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214825 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" containerName="maas-api" Jul 16 21:41:56.214929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214833 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795c4cd1-8471-4d72-a468-844bc9a1eccb" containerName="revoke-keys" Jul 16 21:41:56.214929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214838 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="795c4cd1-8471-4d72-a468-844bc9a1eccb" containerName="revoke-keys" Jul 16 21:41:56.214929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214850 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8151dd96-c41c-43ad-9575-28f3275a3e7f" containerName="revoke-keys" Jul 16 21:41:56.214929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214857 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8151dd96-c41c-43ad-9575-28f3275a3e7f" containerName="revoke-keys" Jul 16 21:41:56.214929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214865 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd9f1a14-ec8a-4b39-8570-2cd667bc2992" containerName="revoke-keys" Jul 16 21:41:56.214929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214870 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd9f1a14-ec8a-4b39-8570-2cd667bc2992" containerName="revoke-keys" Jul 16 21:41:56.214929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214876 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8151dd96-c41c-43ad-9575-28f3275a3e7f" containerName="revoke-keys" Jul 16 21:41:56.214929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214881 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8151dd96-c41c-43ad-9575-28f3275a3e7f" containerName="revoke-keys" Jul 16 21:41:56.214929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214888 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8151dd96-c41c-43ad-9575-28f3275a3e7f" containerName="revoke-keys" Jul 16 21:41:56.214929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214893 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8151dd96-c41c-43ad-9575-28f3275a3e7f" containerName="revoke-keys" Jul 16 21:41:56.214929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214907 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" containerName="maas-api" Jul 16 21:41:56.214929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214912 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" containerName="maas-api" Jul 16 21:41:56.214929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214918 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" containerName="maas-api" Jul 16 21:41:56.214929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214923 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" containerName="maas-api" Jul 16 21:41:56.215541 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214981 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" containerName="maas-api" Jul 16 21:41:56.215541 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214990 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8151dd96-c41c-43ad-9575-28f3275a3e7f" containerName="revoke-keys" Jul 16 21:41:56.215541 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.214995 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8151dd96-c41c-43ad-9575-28f3275a3e7f" containerName="revoke-keys" Jul 16 21:41:56.215541 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.215004 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="795c4cd1-8471-4d72-a468-844bc9a1eccb" containerName="revoke-keys" Jul 16 21:41:56.215541 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.215011 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" containerName="maas-api" Jul 16 21:41:56.215541 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.215016 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" containerName="maas-api" Jul 16 21:41:56.219299 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.219280 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" Jul 16 21:41:56.221864 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.221837 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:41:56.221978 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.221867 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:41:56.228397 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.228369 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm"] Jul 16 21:41:56.319782 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.319745 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5289e1e-d670-410e-ba77-d16a7a84eaa2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm\" (UID: \"e5289e1e-d670-410e-ba77-d16a7a84eaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" Jul 16 21:41:56.420704 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.420652 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5289e1e-d670-410e-ba77-d16a7a84eaa2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm\" (UID: \"e5289e1e-d670-410e-ba77-d16a7a84eaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" Jul 16 21:41:56.421300 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.421281 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5289e1e-d670-410e-ba77-d16a7a84eaa2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm\" (UID: \"e5289e1e-d670-410e-ba77-d16a7a84eaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" Jul 16 21:41:56.531062 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.530940 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" Jul 16 21:41:56.662526 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.662499 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm"] Jul 16 21:41:56.664852 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:41:56.664825 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5289e1e_d670_410e_ba77_d16a7a84eaa2.slice/crio-61b945271a3c8cffe93b09a681e4312cd4f61f5ee5d6efac945fdcd60e8f1900 WatchSource:0}: Error finding container 61b945271a3c8cffe93b09a681e4312cd4f61f5ee5d6efac945fdcd60e8f1900: Status 404 returned error can't find the container with id 61b945271a3c8cffe93b09a681e4312cd4f61f5ee5d6efac945fdcd60e8f1900 Jul 16 21:41:56.920804 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:56.920764 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" event={"ID":"e5289e1e-d670-410e-ba77-d16a7a84eaa2","Type":"ContainerStarted","Data":"61b945271a3c8cffe93b09a681e4312cd4f61f5ee5d6efac945fdcd60e8f1900"} Jul 16 21:41:57.928582 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:57.928541 2577 generic.go:356] "Generic (PLEG): container finished" podID="e5289e1e-d670-410e-ba77-d16a7a84eaa2" containerID="68fc6fe5763a32ab4a857185a33743be7d259447c0080f296172c0ce98a61ba5" exitCode=6 Jul 16 21:41:57.929048 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:57.928668 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" event={"ID":"e5289e1e-d670-410e-ba77-d16a7a84eaa2","Type":"ContainerDied","Data":"68fc6fe5763a32ab4a857185a33743be7d259447c0080f296172c0ce98a61ba5"} Jul 16 21:41:57.929048 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:57.928801 2577 scope.go:117] "RemoveContainer" containerID="68fc6fe5763a32ab4a857185a33743be7d259447c0080f296172c0ce98a61ba5" Jul 16 21:41:58.936484 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:58.936446 2577 generic.go:356] "Generic (PLEG): container finished" podID="e5289e1e-d670-410e-ba77-d16a7a84eaa2" containerID="b0852382727b90144e357f817e3b5542969968d6f5995306ad861e9e6af29a9c" exitCode=6 Jul 16 21:41:58.936890 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:58.936518 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" event={"ID":"e5289e1e-d670-410e-ba77-d16a7a84eaa2","Type":"ContainerDied","Data":"b0852382727b90144e357f817e3b5542969968d6f5995306ad861e9e6af29a9c"} Jul 16 21:41:58.936890 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:58.936569 2577 scope.go:117] "RemoveContainer" containerID="68fc6fe5763a32ab4a857185a33743be7d259447c0080f296172c0ce98a61ba5" Jul 16 21:41:58.936890 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:58.936729 2577 scope.go:117] "RemoveContainer" containerID="b0852382727b90144e357f817e3b5542969968d6f5995306ad861e9e6af29a9c" Jul 16 21:41:58.937059 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:41:58.936988 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm_odh-ai-gateway-infra(e5289e1e-d670-410e-ba77-d16a7a84eaa2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" podUID="e5289e1e-d670-410e-ba77-d16a7a84eaa2" Jul 16 21:41:59.488373 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.488341 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg"] Jul 16 21:41:59.488764 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.488747 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" containerName="maas-api" Jul 16 21:41:59.488860 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.488766 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" containerName="maas-api" Jul 16 21:41:59.488917 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.488858 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8151dd96-c41c-43ad-9575-28f3275a3e7f" containerName="revoke-keys" Jul 16 21:41:59.488917 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.488876 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f49f9ba-1661-404f-ac16-027e4375da41" containerName="maas-api" Jul 16 21:41:59.493061 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.493038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:41:59.496825 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.496806 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f60157c3-serving-cert\"" Jul 16 21:41:59.497317 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.497302 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xcg5c\"" Jul 16 21:41:59.498125 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.498111 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 21:41:59.508284 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.508252 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg"] Jul 16 21:41:59.544678 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.544644 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b759\" (UniqueName: \"kubernetes.io/projected/454a4f80-c2dd-45b4-9b59-9152db8367d1-kube-api-access-4b759\") pod \"maas-api-e2e-ait-f60157c3-54749c956b-dswbg\" (UID: \"454a4f80-c2dd-45b4-9b59-9152db8367d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:41:59.544850 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.544698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454a4f80-c2dd-45b4-9b59-9152db8367d1-openshift-service-ca\") pod \"maas-api-e2e-ait-f60157c3-54749c956b-dswbg\" (UID: \"454a4f80-c2dd-45b4-9b59-9152db8367d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:41:59.544850 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.544724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454a4f80-c2dd-45b4-9b59-9152db8367d1-maas-api-tls\") pod \"maas-api-e2e-ait-f60157c3-54749c956b-dswbg\" (UID: \"454a4f80-c2dd-45b4-9b59-9152db8367d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:41:59.645746 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.645708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b759\" (UniqueName: \"kubernetes.io/projected/454a4f80-c2dd-45b4-9b59-9152db8367d1-kube-api-access-4b759\") pod \"maas-api-e2e-ait-f60157c3-54749c956b-dswbg\" (UID: \"454a4f80-c2dd-45b4-9b59-9152db8367d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:41:59.645954 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.645778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454a4f80-c2dd-45b4-9b59-9152db8367d1-openshift-service-ca\") pod \"maas-api-e2e-ait-f60157c3-54749c956b-dswbg\" (UID: \"454a4f80-c2dd-45b4-9b59-9152db8367d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:41:59.645954 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.645820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454a4f80-c2dd-45b4-9b59-9152db8367d1-maas-api-tls\") pod \"maas-api-e2e-ait-f60157c3-54749c956b-dswbg\" (UID: \"454a4f80-c2dd-45b4-9b59-9152db8367d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:41:59.646083 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:41:59.645983 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-serving-cert: secret "maas-api-e2e-ait-f60157c3-serving-cert" not found Jul 16 21:41:59.646083 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:41:59.646057 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/454a4f80-c2dd-45b4-9b59-9152db8367d1-maas-api-tls podName:454a4f80-c2dd-45b4-9b59-9152db8367d1 nodeName:}" failed. No retries permitted until 2026-07-16 21:42:00.146034373 +0000 UTC m=+1936.321142562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/454a4f80-c2dd-45b4-9b59-9152db8367d1-maas-api-tls") pod "maas-api-e2e-ait-f60157c3-54749c956b-dswbg" (UID: "454a4f80-c2dd-45b4-9b59-9152db8367d1") : secret "maas-api-e2e-ait-f60157c3-serving-cert" not found Jul 16 21:41:59.646508 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.646487 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454a4f80-c2dd-45b4-9b59-9152db8367d1-openshift-service-ca\") pod \"maas-api-e2e-ait-f60157c3-54749c956b-dswbg\" (UID: \"454a4f80-c2dd-45b4-9b59-9152db8367d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:41:59.661755 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.661726 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b759\" (UniqueName: \"kubernetes.io/projected/454a4f80-c2dd-45b4-9b59-9152db8367d1-kube-api-access-4b759\") pod \"maas-api-e2e-ait-f60157c3-54749c956b-dswbg\" (UID: \"454a4f80-c2dd-45b4-9b59-9152db8367d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:41:59.944146 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:41:59.944114 2577 scope.go:117] "RemoveContainer" containerID="b0852382727b90144e357f817e3b5542969968d6f5995306ad861e9e6af29a9c" Jul 16 21:41:59.944568 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:41:59.944342 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm_odh-ai-gateway-infra(e5289e1e-d670-410e-ba77-d16a7a84eaa2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" podUID="e5289e1e-d670-410e-ba77-d16a7a84eaa2" Jul 16 21:42:00.149207 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:00.149141 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454a4f80-c2dd-45b4-9b59-9152db8367d1-maas-api-tls\") pod \"maas-api-e2e-ait-f60157c3-54749c956b-dswbg\" (UID: \"454a4f80-c2dd-45b4-9b59-9152db8367d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:42:00.152251 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:00.152224 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454a4f80-c2dd-45b4-9b59-9152db8367d1-maas-api-tls\") pod \"maas-api-e2e-ait-f60157c3-54749c956b-dswbg\" (UID: \"454a4f80-c2dd-45b4-9b59-9152db8367d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:42:00.403203 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:00.403146 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:42:00.544445 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:00.544411 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg"] Jul 16 21:42:00.545412 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:42:00.545388 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454a4f80_c2dd_45b4_9b59_9152db8367d1.slice/crio-5976bf70d5520380a56a9d1597cd8c50070e057cc7c000f50b32fe58f767c70f WatchSource:0}: Error finding container 5976bf70d5520380a56a9d1597cd8c50070e057cc7c000f50b32fe58f767c70f: Status 404 returned error can't find the container with id 5976bf70d5520380a56a9d1597cd8c50070e057cc7c000f50b32fe58f767c70f Jul 16 21:42:00.950990 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:00.950943 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" event={"ID":"454a4f80-c2dd-45b4-9b59-9152db8367d1","Type":"ContainerStarted","Data":"a98fe5dcdba4dfe44b489c1383ffbd6b0e1467782cab3a4f8f991b9353fb0983"} Jul 16 21:42:00.951465 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:00.951043 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:42:00.951465 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:00.951067 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" event={"ID":"454a4f80-c2dd-45b4-9b59-9152db8367d1","Type":"ContainerStarted","Data":"5976bf70d5520380a56a9d1597cd8c50070e057cc7c000f50b32fe58f767c70f"} Jul 16 21:42:00.972105 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:00.972052 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podStartSLOduration=1.651568843 podStartE2EDuration="1.972035483s" totalImagesPullingTime="320.46664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:42:00.970453281 +0000 UTC m=+1937.145561510" watchObservedRunningTime="2026-07-16 21:42:00.972035483 +0000 UTC m=+1937.147143674" Jul 16 21:42:01.956979 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:01.956945 2577 generic.go:356] "Generic (PLEG): container finished" podID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerID="a98fe5dcdba4dfe44b489c1383ffbd6b0e1467782cab3a4f8f991b9353fb0983" exitCode=1 Jul 16 21:42:01.957389 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:01.957024 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" event={"ID":"454a4f80-c2dd-45b4-9b59-9152db8367d1","Type":"ContainerDied","Data":"a98fe5dcdba4dfe44b489c1383ffbd6b0e1467782cab3a4f8f991b9353fb0983"} Jul 16 21:42:01.957389 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:01.957326 2577 scope.go:117] "RemoveContainer" containerID="a98fe5dcdba4dfe44b489c1383ffbd6b0e1467782cab3a4f8f991b9353fb0983" Jul 16 21:42:02.964552 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:02.964457 2577 generic.go:356] "Generic (PLEG): container finished" podID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerID="144b1f2eaf4de3820936e408741a963aa6043c7393c69d65103fd7f035b3f5cc" exitCode=1 Jul 16 21:42:02.964552 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:02.964511 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" event={"ID":"454a4f80-c2dd-45b4-9b59-9152db8367d1","Type":"ContainerDied","Data":"144b1f2eaf4de3820936e408741a963aa6043c7393c69d65103fd7f035b3f5cc"} Jul 16 21:42:02.965058 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:02.964575 2577 scope.go:117] "RemoveContainer" containerID="a98fe5dcdba4dfe44b489c1383ffbd6b0e1467782cab3a4f8f991b9353fb0983" Jul 16 21:42:02.965058 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:02.964847 2577 scope.go:117] "RemoveContainer" containerID="144b1f2eaf4de3820936e408741a963aa6043c7393c69d65103fd7f035b3f5cc" Jul 16 21:42:02.965162 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:02.965067 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:42:03.974061 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:03.974029 2577 scope.go:117] "RemoveContainer" containerID="144b1f2eaf4de3820936e408741a963aa6043c7393c69d65103fd7f035b3f5cc" Jul 16 21:42:03.974448 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:03.974239 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:42:04.377685 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:04.377650 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w"] Jul 16 21:42:04.382502 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:04.382480 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w" Jul 16 21:42:04.383362 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:04.383337 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9d7b05-16dd-497e-85a2-379cff89bbcd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-77x2w\" (UID: \"0e9d7b05-16dd-497e-85a2-379cff89bbcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w" Jul 16 21:42:04.388986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:04.388922 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w"] Jul 16 21:42:04.483777 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:04.483742 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9d7b05-16dd-497e-85a2-379cff89bbcd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-77x2w\" (UID: \"0e9d7b05-16dd-497e-85a2-379cff89bbcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w" Jul 16 21:42:04.484407 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:04.484389 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9d7b05-16dd-497e-85a2-379cff89bbcd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-77x2w\" (UID: \"0e9d7b05-16dd-497e-85a2-379cff89bbcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w" Jul 16 21:42:04.694119 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:04.694026 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w" Jul 16 21:42:04.829652 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:04.829622 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w"] Jul 16 21:42:04.830968 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:42:04.830943 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9d7b05_16dd_497e_85a2_379cff89bbcd.slice/crio-4fd340fa3153997312c020371c88b3aba418d6b2763131df446abc7d19eeed0f WatchSource:0}: Error finding container 4fd340fa3153997312c020371c88b3aba418d6b2763131df446abc7d19eeed0f: Status 404 returned error can't find the container with id 4fd340fa3153997312c020371c88b3aba418d6b2763131df446abc7d19eeed0f Jul 16 21:42:04.979705 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:04.979618 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w" event={"ID":"0e9d7b05-16dd-497e-85a2-379cff89bbcd","Type":"ContainerStarted","Data":"4fd340fa3153997312c020371c88b3aba418d6b2763131df446abc7d19eeed0f"} Jul 16 21:42:05.951713 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:05.951675 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:42:05.952089 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:05.952074 2577 scope.go:117] "RemoveContainer" containerID="144b1f2eaf4de3820936e408741a963aa6043c7393c69d65103fd7f035b3f5cc" Jul 16 21:42:05.952290 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:05.952270 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:42:05.985833 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:05.985798 2577 generic.go:356] "Generic (PLEG): container finished" podID="0e9d7b05-16dd-497e-85a2-379cff89bbcd" containerID="341ff5cfc116aba4908bc62fb1774bebde1f7cfcc98b3a0e69e0d4c2b0273c14" exitCode=7 Jul 16 21:42:05.986216 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:05.985842 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w" event={"ID":"0e9d7b05-16dd-497e-85a2-379cff89bbcd","Type":"ContainerDied","Data":"341ff5cfc116aba4908bc62fb1774bebde1f7cfcc98b3a0e69e0d4c2b0273c14"} Jul 16 21:42:05.986216 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:05.986054 2577 scope.go:117] "RemoveContainer" containerID="341ff5cfc116aba4908bc62fb1774bebde1f7cfcc98b3a0e69e0d4c2b0273c14" Jul 16 21:42:06.992794 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:06.992757 2577 generic.go:356] "Generic (PLEG): container finished" podID="0e9d7b05-16dd-497e-85a2-379cff89bbcd" containerID="ecf2984812d4a9ab97598d4e13da5bd2cee575cae5b8f8a9db317e07c6069db5" exitCode=7 Jul 16 21:42:06.993214 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:06.992817 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w" event={"ID":"0e9d7b05-16dd-497e-85a2-379cff89bbcd","Type":"ContainerDied","Data":"ecf2984812d4a9ab97598d4e13da5bd2cee575cae5b8f8a9db317e07c6069db5"} Jul 16 21:42:06.993214 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:06.992866 2577 scope.go:117] "RemoveContainer" containerID="341ff5cfc116aba4908bc62fb1774bebde1f7cfcc98b3a0e69e0d4c2b0273c14" Jul 16 21:42:06.993214 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:06.993030 2577 scope.go:117] "RemoveContainer" containerID="ecf2984812d4a9ab97598d4e13da5bd2cee575cae5b8f8a9db317e07c6069db5" Jul 16 21:42:06.993362 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:06.993292 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-77x2w_odh-ai-gateway-infra(0e9d7b05-16dd-497e-85a2-379cff89bbcd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w" podUID="0e9d7b05-16dd-497e-85a2-379cff89bbcd" Jul 16 21:42:07.999760 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:07.999731 2577 scope.go:117] "RemoveContainer" containerID="ecf2984812d4a9ab97598d4e13da5bd2cee575cae5b8f8a9db317e07c6069db5" Jul 16 21:42:08.000207 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:07.999994 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-77x2w_odh-ai-gateway-infra(0e9d7b05-16dd-497e-85a2-379cff89bbcd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w" podUID="0e9d7b05-16dd-497e-85a2-379cff89bbcd" Jul 16 21:42:10.403332 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:10.403288 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:42:10.403827 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:10.403815 2577 scope.go:117] "RemoveContainer" containerID="144b1f2eaf4de3820936e408741a963aa6043c7393c69d65103fd7f035b3f5cc" Jul 16 21:42:10.404042 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:10.404017 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:42:13.444999 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:13.444965 2577 scope.go:117] "RemoveContainer" containerID="b0852382727b90144e357f817e3b5542969968d6f5995306ad861e9e6af29a9c" Jul 16 21:42:14.030223 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:14.030179 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" event={"ID":"e5289e1e-d670-410e-ba77-d16a7a84eaa2","Type":"ContainerStarted","Data":"c8692cffb49fbf1e6de7d4ada88dc18f3ee9ef97de78b213c58414e564d19f1d"} Jul 16 21:42:14.053229 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:14.053156 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" podStartSLOduration=18.05314158 podStartE2EDuration="18.05314158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:42:14.051114838 +0000 UTC m=+1950.226223031" watchObservedRunningTime="2026-07-16 21:42:14.05314158 +0000 UTC m=+1950.228249820" Jul 16 21:42:15.037444 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:15.037408 2577 generic.go:356] "Generic (PLEG): container finished" podID="e5289e1e-d670-410e-ba77-d16a7a84eaa2" containerID="c8692cffb49fbf1e6de7d4ada88dc18f3ee9ef97de78b213c58414e564d19f1d" exitCode=6 Jul 16 21:42:15.037861 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:15.037485 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" event={"ID":"e5289e1e-d670-410e-ba77-d16a7a84eaa2","Type":"ContainerDied","Data":"c8692cffb49fbf1e6de7d4ada88dc18f3ee9ef97de78b213c58414e564d19f1d"} Jul 16 21:42:15.037861 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:15.037550 2577 scope.go:117] "RemoveContainer" containerID="b0852382727b90144e357f817e3b5542969968d6f5995306ad861e9e6af29a9c" Jul 16 21:42:15.037861 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:15.037830 2577 scope.go:117] "RemoveContainer" containerID="c8692cffb49fbf1e6de7d4ada88dc18f3ee9ef97de78b213c58414e564d19f1d" Jul 16 21:42:15.038107 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:15.038088 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm_odh-ai-gateway-infra(e5289e1e-d670-410e-ba77-d16a7a84eaa2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" podUID="e5289e1e-d670-410e-ba77-d16a7a84eaa2" Jul 16 21:42:15.069649 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:15.069615 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm"] Jul 16 21:42:16.179941 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:16.179916 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" Jul 16 21:42:16.286599 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:16.286570 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5289e1e-d670-410e-ba77-d16a7a84eaa2-maas-api-service-ca\") pod \"e5289e1e-d670-410e-ba77-d16a7a84eaa2\" (UID: \"e5289e1e-d670-410e-ba77-d16a7a84eaa2\") " Jul 16 21:42:16.286933 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:16.286906 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5289e1e-d670-410e-ba77-d16a7a84eaa2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5289e1e-d670-410e-ba77-d16a7a84eaa2" (UID: "e5289e1e-d670-410e-ba77-d16a7a84eaa2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:42:16.387789 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:16.387750 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5289e1e-d670-410e-ba77-d16a7a84eaa2-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:42:17.051660 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:17.051623 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" event={"ID":"e5289e1e-d670-410e-ba77-d16a7a84eaa2","Type":"ContainerDied","Data":"61b945271a3c8cffe93b09a681e4312cd4f61f5ee5d6efac945fdcd60e8f1900"} Jul 16 21:42:17.051660 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:17.051647 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm" Jul 16 21:42:17.051925 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:17.051691 2577 scope.go:117] "RemoveContainer" containerID="c8692cffb49fbf1e6de7d4ada88dc18f3ee9ef97de78b213c58414e564d19f1d" Jul 16 21:42:17.072540 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:17.072513 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm"] Jul 16 21:42:17.080999 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:17.080968 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-j5nkm"] Jul 16 21:42:18.455856 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:18.455811 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5289e1e-d670-410e-ba77-d16a7a84eaa2" path="/var/lib/kubelet/pods/e5289e1e-d670-410e-ba77-d16a7a84eaa2/volumes" Jul 16 21:42:21.444995 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:21.444962 2577 scope.go:117] "RemoveContainer" containerID="144b1f2eaf4de3820936e408741a963aa6043c7393c69d65103fd7f035b3f5cc" Jul 16 21:42:22.080612 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:22.080514 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" event={"ID":"454a4f80-c2dd-45b4-9b59-9152db8367d1","Type":"ContainerStarted","Data":"382a7b20cd823eecd55f9b0a39c7b306534976bf8ffcd980f366ad07fadb65cf"} Jul 16 21:42:22.080797 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:22.080748 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:42:23.087721 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:23.087689 2577 generic.go:356] "Generic (PLEG): container finished" podID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerID="382a7b20cd823eecd55f9b0a39c7b306534976bf8ffcd980f366ad07fadb65cf" exitCode=1 Jul 16 21:42:23.088148 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:23.087751 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" event={"ID":"454a4f80-c2dd-45b4-9b59-9152db8367d1","Type":"ContainerDied","Data":"382a7b20cd823eecd55f9b0a39c7b306534976bf8ffcd980f366ad07fadb65cf"} Jul 16 21:42:23.088148 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:23.087798 2577 scope.go:117] "RemoveContainer" containerID="144b1f2eaf4de3820936e408741a963aa6043c7393c69d65103fd7f035b3f5cc" Jul 16 21:42:23.088148 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:23.088015 2577 scope.go:117] "RemoveContainer" containerID="382a7b20cd823eecd55f9b0a39c7b306534976bf8ffcd980f366ad07fadb65cf" Jul 16 21:42:23.088349 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:23.088328 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:42:23.445538 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:23.445491 2577 scope.go:117] "RemoveContainer" containerID="ecf2984812d4a9ab97598d4e13da5bd2cee575cae5b8f8a9db317e07c6069db5" Jul 16 21:42:24.095241 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:24.095216 2577 scope.go:117] "RemoveContainer" containerID="382a7b20cd823eecd55f9b0a39c7b306534976bf8ffcd980f366ad07fadb65cf" Jul 16 21:42:24.095634 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:24.095481 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:42:24.096826 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:24.096803 2577 generic.go:356] "Generic (PLEG): container finished" podID="0e9d7b05-16dd-497e-85a2-379cff89bbcd" containerID="0c8a60bddcc2c6de63f0b9fd051255a47435ccf5174aff1c389e9307e81586b5" exitCode=7 Jul 16 21:42:24.096949 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:24.096834 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w" event={"ID":"0e9d7b05-16dd-497e-85a2-379cff89bbcd","Type":"ContainerDied","Data":"0c8a60bddcc2c6de63f0b9fd051255a47435ccf5174aff1c389e9307e81586b5"} Jul 16 21:42:24.096949 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:24.096876 2577 scope.go:117] "RemoveContainer" containerID="ecf2984812d4a9ab97598d4e13da5bd2cee575cae5b8f8a9db317e07c6069db5" Jul 16 21:42:24.097166 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:24.097150 2577 scope.go:117] "RemoveContainer" containerID="0c8a60bddcc2c6de63f0b9fd051255a47435ccf5174aff1c389e9307e81586b5" Jul 16 21:42:24.097379 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:24.097353 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-77x2w_odh-ai-gateway-infra(0e9d7b05-16dd-497e-85a2-379cff89bbcd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w" podUID="0e9d7b05-16dd-497e-85a2-379cff89bbcd" Jul 16 21:42:25.082623 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.082585 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t"] Jul 16 21:42:25.082944 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.082932 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5289e1e-d670-410e-ba77-d16a7a84eaa2" containerName="revoke-keys" Jul 16 21:42:25.082944 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.082945 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5289e1e-d670-410e-ba77-d16a7a84eaa2" containerName="revoke-keys" Jul 16 21:42:25.083042 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.082961 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5289e1e-d670-410e-ba77-d16a7a84eaa2" containerName="revoke-keys" Jul 16 21:42:25.083042 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.082968 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5289e1e-d670-410e-ba77-d16a7a84eaa2" containerName="revoke-keys" Jul 16 21:42:25.083106 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.083041 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5289e1e-d670-410e-ba77-d16a7a84eaa2" containerName="revoke-keys" Jul 16 21:42:25.083106 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.083052 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5289e1e-d670-410e-ba77-d16a7a84eaa2" containerName="revoke-keys" Jul 16 21:42:25.087215 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.087171 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t" Jul 16 21:42:25.096986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.096958 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t"] Jul 16 21:42:25.144914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.144878 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w"] Jul 16 21:42:25.166691 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.166649 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe326d3-5072-497b-98d9-181da8983a3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t\" (UID: \"5fe326d3-5072-497b-98d9-181da8983a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t" Jul 16 21:42:25.267896 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.267863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe326d3-5072-497b-98d9-181da8983a3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t\" (UID: \"5fe326d3-5072-497b-98d9-181da8983a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t" Jul 16 21:42:25.268718 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.268695 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe326d3-5072-497b-98d9-181da8983a3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t\" (UID: \"5fe326d3-5072-497b-98d9-181da8983a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t" Jul 16 21:42:25.275461 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.275439 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w" Jul 16 21:42:25.368525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.368436 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9d7b05-16dd-497e-85a2-379cff89bbcd-maas-api-service-ca\") pod \"0e9d7b05-16dd-497e-85a2-379cff89bbcd\" (UID: \"0e9d7b05-16dd-497e-85a2-379cff89bbcd\") " Jul 16 21:42:25.368839 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.368817 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9d7b05-16dd-497e-85a2-379cff89bbcd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e9d7b05-16dd-497e-85a2-379cff89bbcd" (UID: "0e9d7b05-16dd-497e-85a2-379cff89bbcd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:42:25.399391 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.399360 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t" Jul 16 21:42:25.470069 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.470044 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9d7b05-16dd-497e-85a2-379cff89bbcd-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:42:25.535299 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:25.535268 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t"] Jul 16 21:42:25.536698 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:42:25.536670 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe326d3_5072_497b_98d9_181da8983a3e.slice/crio-1e70836aadf5a248e9d9fe2ba43792d99b84f3ee5a141deb8b8a18fe1f1b1abd WatchSource:0}: Error finding container 1e70836aadf5a248e9d9fe2ba43792d99b84f3ee5a141deb8b8a18fe1f1b1abd: Status 404 returned error can't find the container with id 1e70836aadf5a248e9d9fe2ba43792d99b84f3ee5a141deb8b8a18fe1f1b1abd Jul 16 21:42:26.110031 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:26.109997 2577 generic.go:356] "Generic (PLEG): container finished" podID="5fe326d3-5072-497b-98d9-181da8983a3e" containerID="66845a437f2e0a28887cff300f6718f98f3e18c67877fb42b5ec91c5edd62dbd" exitCode=6 Jul 16 21:42:26.110477 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:26.110073 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t" event={"ID":"5fe326d3-5072-497b-98d9-181da8983a3e","Type":"ContainerDied","Data":"66845a437f2e0a28887cff300f6718f98f3e18c67877fb42b5ec91c5edd62dbd"} Jul 16 21:42:26.110477 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:26.110127 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t" event={"ID":"5fe326d3-5072-497b-98d9-181da8983a3e","Type":"ContainerStarted","Data":"1e70836aadf5a248e9d9fe2ba43792d99b84f3ee5a141deb8b8a18fe1f1b1abd"} Jul 16 21:42:26.110477 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:26.110229 2577 scope.go:117] "RemoveContainer" containerID="66845a437f2e0a28887cff300f6718f98f3e18c67877fb42b5ec91c5edd62dbd" Jul 16 21:42:26.111474 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:26.111453 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w" event={"ID":"0e9d7b05-16dd-497e-85a2-379cff89bbcd","Type":"ContainerDied","Data":"4fd340fa3153997312c020371c88b3aba418d6b2763131df446abc7d19eeed0f"} Jul 16 21:42:26.111474 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:26.111473 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w" Jul 16 21:42:26.111609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:26.111509 2577 scope.go:117] "RemoveContainer" containerID="0c8a60bddcc2c6de63f0b9fd051255a47435ccf5174aff1c389e9307e81586b5" Jul 16 21:42:26.194152 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:26.194115 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w"] Jul 16 21:42:26.201759 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:26.201731 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-77x2w"] Jul 16 21:42:26.450675 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:26.450641 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e9d7b05-16dd-497e-85a2-379cff89bbcd" path="/var/lib/kubelet/pods/0e9d7b05-16dd-497e-85a2-379cff89bbcd/volumes" Jul 16 21:42:27.118476 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:27.118389 2577 generic.go:356] "Generic (PLEG): container finished" podID="5fe326d3-5072-497b-98d9-181da8983a3e" containerID="be2b134eefc4a9fc1433cc0d66668f6a6fd46e27d59bc98c282dbe3a6f3e2bc1" exitCode=6 Jul 16 21:42:27.118898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:27.118475 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t" event={"ID":"5fe326d3-5072-497b-98d9-181da8983a3e","Type":"ContainerDied","Data":"be2b134eefc4a9fc1433cc0d66668f6a6fd46e27d59bc98c282dbe3a6f3e2bc1"} Jul 16 21:42:27.118898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:27.118532 2577 scope.go:117] "RemoveContainer" containerID="66845a437f2e0a28887cff300f6718f98f3e18c67877fb42b5ec91c5edd62dbd" Jul 16 21:42:27.118898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:27.118673 2577 scope.go:117] "RemoveContainer" containerID="be2b134eefc4a9fc1433cc0d66668f6a6fd46e27d59bc98c282dbe3a6f3e2bc1" Jul 16 21:42:27.119059 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:27.118942 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t_odh-ai-gateway-infra(5fe326d3-5072-497b-98d9-181da8983a3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t" podUID="5fe326d3-5072-497b-98d9-181da8983a3e" Jul 16 21:42:28.126927 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:28.126896 2577 scope.go:117] "RemoveContainer" containerID="be2b134eefc4a9fc1433cc0d66668f6a6fd46e27d59bc98c282dbe3a6f3e2bc1" Jul 16 21:42:28.127346 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:28.127117 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t_odh-ai-gateway-infra(5fe326d3-5072-497b-98d9-181da8983a3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t" podUID="5fe326d3-5072-497b-98d9-181da8983a3e" Jul 16 21:42:30.404230 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:30.404168 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:42:30.404628 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:30.404618 2577 scope.go:117] "RemoveContainer" containerID="382a7b20cd823eecd55f9b0a39c7b306534976bf8ffcd980f366ad07fadb65cf" Jul 16 21:42:30.404828 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:30.404807 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:42:41.445590 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:41.445510 2577 scope.go:117] "RemoveContainer" containerID="382a7b20cd823eecd55f9b0a39c7b306534976bf8ffcd980f366ad07fadb65cf" Jul 16 21:42:41.447919 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:41.445721 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:42:42.445688 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:42.445646 2577 scope.go:117] "RemoveContainer" containerID="be2b134eefc4a9fc1433cc0d66668f6a6fd46e27d59bc98c282dbe3a6f3e2bc1" Jul 16 21:42:43.205531 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:43.205498 2577 generic.go:356] "Generic (PLEG): container finished" podID="5fe326d3-5072-497b-98d9-181da8983a3e" containerID="04bf01230bf4027a3e1ebeb6591d2335fe94d8ce75925defb51d9f6c1b7a9423" exitCode=6 Jul 16 21:42:43.205531 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:43.205542 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t" event={"ID":"5fe326d3-5072-497b-98d9-181da8983a3e","Type":"ContainerDied","Data":"04bf01230bf4027a3e1ebeb6591d2335fe94d8ce75925defb51d9f6c1b7a9423"} Jul 16 21:42:43.205818 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:43.205592 2577 scope.go:117] "RemoveContainer" containerID="be2b134eefc4a9fc1433cc0d66668f6a6fd46e27d59bc98c282dbe3a6f3e2bc1" Jul 16 21:42:43.205911 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:43.205893 2577 scope.go:117] "RemoveContainer" containerID="04bf01230bf4027a3e1ebeb6591d2335fe94d8ce75925defb51d9f6c1b7a9423" Jul 16 21:42:43.206145 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:43.206123 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t_odh-ai-gateway-infra(5fe326d3-5072-497b-98d9-181da8983a3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t" podUID="5fe326d3-5072-497b-98d9-181da8983a3e" Jul 16 21:42:44.233682 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:44.233646 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t"] Jul 16 21:42:44.375734 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:44.375709 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t" Jul 16 21:42:44.545377 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:44.545345 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe326d3-5072-497b-98d9-181da8983a3e-maas-api-service-ca\") pod \"5fe326d3-5072-497b-98d9-181da8983a3e\" (UID: \"5fe326d3-5072-497b-98d9-181da8983a3e\") " Jul 16 21:42:44.545738 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:44.545711 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe326d3-5072-497b-98d9-181da8983a3e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5fe326d3-5072-497b-98d9-181da8983a3e" (UID: "5fe326d3-5072-497b-98d9-181da8983a3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:42:44.646334 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:44.646292 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe326d3-5072-497b-98d9-181da8983a3e-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:42:45.219173 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:45.219145 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t" Jul 16 21:42:45.219364 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:45.219204 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t" event={"ID":"5fe326d3-5072-497b-98d9-181da8983a3e","Type":"ContainerDied","Data":"1e70836aadf5a248e9d9fe2ba43792d99b84f3ee5a141deb8b8a18fe1f1b1abd"} Jul 16 21:42:45.219364 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:45.219270 2577 scope.go:117] "RemoveContainer" containerID="04bf01230bf4027a3e1ebeb6591d2335fe94d8ce75925defb51d9f6c1b7a9423" Jul 16 21:42:45.258699 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:45.258667 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t"] Jul 16 21:42:45.271255 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:45.271224 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-9cz5t"] Jul 16 21:42:46.450734 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:46.450693 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fe326d3-5072-497b-98d9-181da8983a3e" path="/var/lib/kubelet/pods/5fe326d3-5072-497b-98d9-181da8983a3e/volumes" Jul 16 21:42:52.445650 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:52.445621 2577 scope.go:117] "RemoveContainer" containerID="382a7b20cd823eecd55f9b0a39c7b306534976bf8ffcd980f366ad07fadb65cf" Jul 16 21:42:52.446035 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:52.445917 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:42:53.261922 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:53.261893 2577 generic.go:356] "Generic (PLEG): container finished" podID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerID="3da48d7c9a6ac95a50f22924eaa7c344386bd9dba06c700d039a2dc6e8e6db95" exitCode=1 Jul 16 21:42:53.262080 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:53.261967 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" event={"ID":"454a4f80-c2dd-45b4-9b59-9152db8367d1","Type":"ContainerDied","Data":"3da48d7c9a6ac95a50f22924eaa7c344386bd9dba06c700d039a2dc6e8e6db95"} Jul 16 21:42:53.262080 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:53.262035 2577 scope.go:117] "RemoveContainer" containerID="382a7b20cd823eecd55f9b0a39c7b306534976bf8ffcd980f366ad07fadb65cf" Jul 16 21:42:53.262423 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:53.262408 2577 scope.go:117] "RemoveContainer" containerID="3da48d7c9a6ac95a50f22924eaa7c344386bd9dba06c700d039a2dc6e8e6db95" Jul 16 21:42:53.262606 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:53.262584 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:42:56.292829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.292784 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4"] Jul 16 21:42:56.293352 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293331 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe326d3-5072-497b-98d9-181da8983a3e" containerName="revoke-keys" Jul 16 21:42:56.293448 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293356 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe326d3-5072-497b-98d9-181da8983a3e" containerName="revoke-keys" Jul 16 21:42:56.293448 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293376 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe326d3-5072-497b-98d9-181da8983a3e" containerName="revoke-keys" Jul 16 21:42:56.293448 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293385 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe326d3-5072-497b-98d9-181da8983a3e" containerName="revoke-keys" Jul 16 21:42:56.293448 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293397 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e9d7b05-16dd-497e-85a2-379cff89bbcd" containerName="revoke-keys" Jul 16 21:42:56.293448 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293406 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e9d7b05-16dd-497e-85a2-379cff89bbcd" containerName="revoke-keys" Jul 16 21:42:56.293448 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293414 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e9d7b05-16dd-497e-85a2-379cff89bbcd" containerName="revoke-keys" Jul 16 21:42:56.293448 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293422 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e9d7b05-16dd-497e-85a2-379cff89bbcd" containerName="revoke-keys" Jul 16 21:42:56.293448 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293434 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe326d3-5072-497b-98d9-181da8983a3e" containerName="revoke-keys" Jul 16 21:42:56.293448 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293442 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe326d3-5072-497b-98d9-181da8983a3e" containerName="revoke-keys" Jul 16 21:42:56.293886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293464 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5289e1e-d670-410e-ba77-d16a7a84eaa2" containerName="revoke-keys" Jul 16 21:42:56.293886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293472 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5289e1e-d670-410e-ba77-d16a7a84eaa2" containerName="revoke-keys" Jul 16 21:42:56.293886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293493 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e9d7b05-16dd-497e-85a2-379cff89bbcd" containerName="revoke-keys" Jul 16 21:42:56.293886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293501 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e9d7b05-16dd-497e-85a2-379cff89bbcd" containerName="revoke-keys" Jul 16 21:42:56.293886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293592 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe326d3-5072-497b-98d9-181da8983a3e" containerName="revoke-keys" Jul 16 21:42:56.293886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293606 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe326d3-5072-497b-98d9-181da8983a3e" containerName="revoke-keys" Jul 16 21:42:56.293886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293616 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e9d7b05-16dd-497e-85a2-379cff89bbcd" containerName="revoke-keys" Jul 16 21:42:56.293886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293628 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e9d7b05-16dd-497e-85a2-379cff89bbcd" containerName="revoke-keys" Jul 16 21:42:56.293886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293637 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5289e1e-d670-410e-ba77-d16a7a84eaa2" containerName="revoke-keys" Jul 16 21:42:56.293886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.293647 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e9d7b05-16dd-497e-85a2-379cff89bbcd" containerName="revoke-keys" Jul 16 21:42:56.297937 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.297914 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" Jul 16 21:42:56.301014 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.300988 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:42:56.306249 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.305675 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4"] Jul 16 21:42:56.344093 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.344063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2dc4b8f-806a-4993-9a33-6f24e96c0ef0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4\" (UID: \"a2dc4b8f-806a-4993-9a33-6f24e96c0ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" Jul 16 21:42:56.444556 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.444513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2dc4b8f-806a-4993-9a33-6f24e96c0ef0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4\" (UID: \"a2dc4b8f-806a-4993-9a33-6f24e96c0ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" Jul 16 21:42:56.445170 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.445152 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2dc4b8f-806a-4993-9a33-6f24e96c0ef0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4\" (UID: \"a2dc4b8f-806a-4993-9a33-6f24e96c0ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" Jul 16 21:42:56.610734 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.610644 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" Jul 16 21:42:56.750511 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:56.750488 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4"] Jul 16 21:42:56.752348 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:42:56.752322 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2dc4b8f_806a_4993_9a33_6f24e96c0ef0.slice/crio-54450555249ab0ed991ba4832029d754d633deb444d9ffe2a81ad06cf4ae5973 WatchSource:0}: Error finding container 54450555249ab0ed991ba4832029d754d633deb444d9ffe2a81ad06cf4ae5973: Status 404 returned error can't find the container with id 54450555249ab0ed991ba4832029d754d633deb444d9ffe2a81ad06cf4ae5973 Jul 16 21:42:57.081417 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:57.081387 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:42:57.081757 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:57.081743 2577 scope.go:117] "RemoveContainer" containerID="3da48d7c9a6ac95a50f22924eaa7c344386bd9dba06c700d039a2dc6e8e6db95" Jul 16 21:42:57.081960 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:57.081937 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:42:57.289985 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:57.289893 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" event={"ID":"a2dc4b8f-806a-4993-9a33-6f24e96c0ef0","Type":"ContainerStarted","Data":"d42d9d56321d35c5f4ec48373e8c602dd51a7d0fe527e258c9700c0e0fe0f13b"} Jul 16 21:42:57.289985 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:57.289948 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" event={"ID":"a2dc4b8f-806a-4993-9a33-6f24e96c0ef0","Type":"ContainerStarted","Data":"54450555249ab0ed991ba4832029d754d633deb444d9ffe2a81ad06cf4ae5973"} Jul 16 21:42:57.308653 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:57.308594 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" podStartSLOduration=1.308578156 podStartE2EDuration="1.308578156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:42:57.306422319 +0000 UTC m=+1993.481530526" watchObservedRunningTime="2026-07-16 21:42:57.308578156 +0000 UTC m=+1993.483686394" Jul 16 21:42:58.296358 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:58.296320 2577 generic.go:356] "Generic (PLEG): container finished" podID="a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" containerID="d42d9d56321d35c5f4ec48373e8c602dd51a7d0fe527e258c9700c0e0fe0f13b" exitCode=6 Jul 16 21:42:58.296575 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:58.296404 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" event={"ID":"a2dc4b8f-806a-4993-9a33-6f24e96c0ef0","Type":"ContainerDied","Data":"d42d9d56321d35c5f4ec48373e8c602dd51a7d0fe527e258c9700c0e0fe0f13b"} Jul 16 21:42:58.296645 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:58.296622 2577 scope.go:117] "RemoveContainer" containerID="d42d9d56321d35c5f4ec48373e8c602dd51a7d0fe527e258c9700c0e0fe0f13b" Jul 16 21:42:59.303481 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:59.303385 2577 generic.go:356] "Generic (PLEG): container finished" podID="a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" containerID="84a6f409eb7933d99686cb9c85ddb70058df66bf29e63f37a1a408c9a5c40546" exitCode=6 Jul 16 21:42:59.303481 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:59.303452 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" event={"ID":"a2dc4b8f-806a-4993-9a33-6f24e96c0ef0","Type":"ContainerDied","Data":"84a6f409eb7933d99686cb9c85ddb70058df66bf29e63f37a1a408c9a5c40546"} Jul 16 21:42:59.303972 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:59.303513 2577 scope.go:117] "RemoveContainer" containerID="d42d9d56321d35c5f4ec48373e8c602dd51a7d0fe527e258c9700c0e0fe0f13b" Jul 16 21:42:59.303972 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:42:59.303672 2577 scope.go:117] "RemoveContainer" containerID="84a6f409eb7933d99686cb9c85ddb70058df66bf29e63f37a1a408c9a5c40546" Jul 16 21:42:59.303972 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:42:59.303942 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4_odh-ai-gateway-infra(a2dc4b8f-806a-4993-9a33-6f24e96c0ef0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" podUID="a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" Jul 16 21:43:00.309828 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:00.309794 2577 scope.go:117] "RemoveContainer" containerID="84a6f409eb7933d99686cb9c85ddb70058df66bf29e63f37a1a408c9a5c40546" Jul 16 21:43:00.310358 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:00.310034 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4_odh-ai-gateway-infra(a2dc4b8f-806a-4993-9a33-6f24e96c0ef0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" podUID="a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" Jul 16 21:43:00.404206 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:00.404145 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:43:00.404610 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:00.404595 2577 scope.go:117] "RemoveContainer" containerID="3da48d7c9a6ac95a50f22924eaa7c344386bd9dba06c700d039a2dc6e8e6db95" Jul 16 21:43:00.404799 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:00.404779 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:43:04.443719 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:04.443679 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn"] Jul 16 21:43:04.444211 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:04.444094 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe326d3-5072-497b-98d9-181da8983a3e" containerName="revoke-keys" Jul 16 21:43:04.448370 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:04.448347 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" Jul 16 21:43:04.456347 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:04.456317 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn"] Jul 16 21:43:04.513055 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:04.513013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eeac645-908f-4137-a50d-57a686d04f9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn\" (UID: \"9eeac645-908f-4137-a50d-57a686d04f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" Jul 16 21:43:04.614712 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:04.614667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eeac645-908f-4137-a50d-57a686d04f9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn\" (UID: \"9eeac645-908f-4137-a50d-57a686d04f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" Jul 16 21:43:04.615344 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:04.615325 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eeac645-908f-4137-a50d-57a686d04f9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn\" (UID: \"9eeac645-908f-4137-a50d-57a686d04f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" Jul 16 21:43:04.760623 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:04.760530 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" Jul 16 21:43:04.895921 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:04.895893 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn"] Jul 16 21:43:04.897717 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:43:04.897689 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eeac645_908f_4137_a50d_57a686d04f9e.slice/crio-d9787986a753284ee61d0abc0163fe5fe7a139c539e978361fa1bc7f4a1340dd WatchSource:0}: Error finding container d9787986a753284ee61d0abc0163fe5fe7a139c539e978361fa1bc7f4a1340dd: Status 404 returned error can't find the container with id d9787986a753284ee61d0abc0163fe5fe7a139c539e978361fa1bc7f4a1340dd Jul 16 21:43:05.337519 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:05.337422 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" event={"ID":"9eeac645-908f-4137-a50d-57a686d04f9e","Type":"ContainerStarted","Data":"6f7d89418372aa10fc37f5bf3cb27ac7f1c3f33fb93d17e86914b50fd3035017"} Jul 16 21:43:05.337672 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:05.337521 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" event={"ID":"9eeac645-908f-4137-a50d-57a686d04f9e","Type":"ContainerStarted","Data":"d9787986a753284ee61d0abc0163fe5fe7a139c539e978361fa1bc7f4a1340dd"} Jul 16 21:43:05.353640 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:05.353586 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" podStartSLOduration=1.353572485 podStartE2EDuration="1.353572485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:43:05.352806269 +0000 UTC m=+2001.527914462" watchObservedRunningTime="2026-07-16 21:43:05.353572485 +0000 UTC m=+2001.528680676" Jul 16 21:43:06.344591 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:06.344555 2577 generic.go:356] "Generic (PLEG): container finished" podID="9eeac645-908f-4137-a50d-57a686d04f9e" containerID="6f7d89418372aa10fc37f5bf3cb27ac7f1c3f33fb93d17e86914b50fd3035017" exitCode=7 Jul 16 21:43:06.344981 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:06.344608 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" event={"ID":"9eeac645-908f-4137-a50d-57a686d04f9e","Type":"ContainerDied","Data":"6f7d89418372aa10fc37f5bf3cb27ac7f1c3f33fb93d17e86914b50fd3035017"} Jul 16 21:43:06.344981 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:06.344788 2577 scope.go:117] "RemoveContainer" containerID="6f7d89418372aa10fc37f5bf3cb27ac7f1c3f33fb93d17e86914b50fd3035017" Jul 16 21:43:07.351723 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:07.351640 2577 generic.go:356] "Generic (PLEG): container finished" podID="9eeac645-908f-4137-a50d-57a686d04f9e" containerID="e616d670e28680e4ef3c7d4cab3b0dca667438ad35994d8f091c5ed2eb0af4f2" exitCode=7 Jul 16 21:43:07.351723 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:07.351712 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" event={"ID":"9eeac645-908f-4137-a50d-57a686d04f9e","Type":"ContainerDied","Data":"e616d670e28680e4ef3c7d4cab3b0dca667438ad35994d8f091c5ed2eb0af4f2"} Jul 16 21:43:07.352170 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:07.351755 2577 scope.go:117] "RemoveContainer" containerID="6f7d89418372aa10fc37f5bf3cb27ac7f1c3f33fb93d17e86914b50fd3035017" Jul 16 21:43:07.352170 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:07.351954 2577 scope.go:117] "RemoveContainer" containerID="e616d670e28680e4ef3c7d4cab3b0dca667438ad35994d8f091c5ed2eb0af4f2" Jul 16 21:43:07.352293 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:07.352209 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn_odh-ai-gateway-infra(9eeac645-908f-4137-a50d-57a686d04f9e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" podUID="9eeac645-908f-4137-a50d-57a686d04f9e" Jul 16 21:43:08.358489 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:08.358466 2577 scope.go:117] "RemoveContainer" containerID="e616d670e28680e4ef3c7d4cab3b0dca667438ad35994d8f091c5ed2eb0af4f2" Jul 16 21:43:08.358872 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:08.358654 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn_odh-ai-gateway-infra(9eeac645-908f-4137-a50d-57a686d04f9e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" podUID="9eeac645-908f-4137-a50d-57a686d04f9e" Jul 16 21:43:11.445671 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:11.445639 2577 scope.go:117] "RemoveContainer" containerID="3da48d7c9a6ac95a50f22924eaa7c344386bd9dba06c700d039a2dc6e8e6db95" Jul 16 21:43:11.446055 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:11.445791 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:43:13.445640 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:13.445603 2577 scope.go:117] "RemoveContainer" containerID="84a6f409eb7933d99686cb9c85ddb70058df66bf29e63f37a1a408c9a5c40546" Jul 16 21:43:14.391907 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:14.391877 2577 generic.go:356] "Generic (PLEG): container finished" podID="a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" containerID="a7e204b1625b8df83b97c4505bcb20a0303e2628c7a290dea8310519a772ee14" exitCode=6 Jul 16 21:43:14.392067 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:14.391949 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" event={"ID":"a2dc4b8f-806a-4993-9a33-6f24e96c0ef0","Type":"ContainerDied","Data":"a7e204b1625b8df83b97c4505bcb20a0303e2628c7a290dea8310519a772ee14"} Jul 16 21:43:14.392067 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:14.392008 2577 scope.go:117] "RemoveContainer" containerID="84a6f409eb7933d99686cb9c85ddb70058df66bf29e63f37a1a408c9a5c40546" Jul 16 21:43:14.392419 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:14.392403 2577 scope.go:117] "RemoveContainer" containerID="a7e204b1625b8df83b97c4505bcb20a0303e2628c7a290dea8310519a772ee14" Jul 16 21:43:14.392611 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:14.392591 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4_odh-ai-gateway-infra(a2dc4b8f-806a-4993-9a33-6f24e96c0ef0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" podUID="a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" Jul 16 21:43:15.420740 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:15.420699 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4"] Jul 16 21:43:15.555276 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:15.555251 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" Jul 16 21:43:15.612972 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:15.612942 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2dc4b8f-806a-4993-9a33-6f24e96c0ef0-maas-api-service-ca\") pod \"a2dc4b8f-806a-4993-9a33-6f24e96c0ef0\" (UID: \"a2dc4b8f-806a-4993-9a33-6f24e96c0ef0\") " Jul 16 21:43:15.613316 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:15.613293 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2dc4b8f-806a-4993-9a33-6f24e96c0ef0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" (UID: "a2dc4b8f-806a-4993-9a33-6f24e96c0ef0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:43:15.714535 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:15.714456 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2dc4b8f-806a-4993-9a33-6f24e96c0ef0-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:43:16.404675 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:16.404648 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" Jul 16 21:43:16.404855 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:16.404641 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4" event={"ID":"a2dc4b8f-806a-4993-9a33-6f24e96c0ef0","Type":"ContainerDied","Data":"54450555249ab0ed991ba4832029d754d633deb444d9ffe2a81ad06cf4ae5973"} Jul 16 21:43:16.404855 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:16.404787 2577 scope.go:117] "RemoveContainer" containerID="a7e204b1625b8df83b97c4505bcb20a0303e2628c7a290dea8310519a772ee14" Jul 16 21:43:16.434078 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:16.434051 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4"] Jul 16 21:43:16.452899 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:16.452872 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vk5r4"] Jul 16 21:43:18.450898 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:18.450862 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" path="/var/lib/kubelet/pods/a2dc4b8f-806a-4993-9a33-6f24e96c0ef0/volumes" Jul 16 21:43:21.445068 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:21.445038 2577 scope.go:117] "RemoveContainer" containerID="e616d670e28680e4ef3c7d4cab3b0dca667438ad35994d8f091c5ed2eb0af4f2" Jul 16 21:43:22.437460 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:22.437427 2577 generic.go:356] "Generic (PLEG): container finished" podID="9eeac645-908f-4137-a50d-57a686d04f9e" containerID="1671535d02a4611649b7f3c5ad55aad07bec61f54e7d732275a3e72371174f88" exitCode=7 Jul 16 21:43:22.437607 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:22.437499 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" event={"ID":"9eeac645-908f-4137-a50d-57a686d04f9e","Type":"ContainerDied","Data":"1671535d02a4611649b7f3c5ad55aad07bec61f54e7d732275a3e72371174f88"} Jul 16 21:43:22.437607 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:22.437559 2577 scope.go:117] "RemoveContainer" containerID="e616d670e28680e4ef3c7d4cab3b0dca667438ad35994d8f091c5ed2eb0af4f2" Jul 16 21:43:22.437844 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:22.437824 2577 scope.go:117] "RemoveContainer" containerID="1671535d02a4611649b7f3c5ad55aad07bec61f54e7d732275a3e72371174f88" Jul 16 21:43:22.438054 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:22.438032 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn_odh-ai-gateway-infra(9eeac645-908f-4137-a50d-57a686d04f9e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" podUID="9eeac645-908f-4137-a50d-57a686d04f9e" Jul 16 21:43:23.476640 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:23.476606 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn"] Jul 16 21:43:23.604262 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:23.604233 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" Jul 16 21:43:23.690224 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:23.690170 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eeac645-908f-4137-a50d-57a686d04f9e-maas-api-service-ca\") pod \"9eeac645-908f-4137-a50d-57a686d04f9e\" (UID: \"9eeac645-908f-4137-a50d-57a686d04f9e\") " Jul 16 21:43:23.690530 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:23.690507 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eeac645-908f-4137-a50d-57a686d04f9e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9eeac645-908f-4137-a50d-57a686d04f9e" (UID: "9eeac645-908f-4137-a50d-57a686d04f9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:43:23.791836 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:23.791757 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eeac645-908f-4137-a50d-57a686d04f9e-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:43:24.447982 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:24.447954 2577 scope.go:117] "RemoveContainer" containerID="3da48d7c9a6ac95a50f22924eaa7c344386bd9dba06c700d039a2dc6e8e6db95" Jul 16 21:43:24.448202 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:24.448166 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:43:24.450855 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:24.450832 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" event={"ID":"9eeac645-908f-4137-a50d-57a686d04f9e","Type":"ContainerDied","Data":"d9787986a753284ee61d0abc0163fe5fe7a139c539e978361fa1bc7f4a1340dd"} Jul 16 21:43:24.450970 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:24.450864 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn" Jul 16 21:43:24.450970 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:24.450885 2577 scope.go:117] "RemoveContainer" containerID="1671535d02a4611649b7f3c5ad55aad07bec61f54e7d732275a3e72371174f88" Jul 16 21:43:24.475250 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:24.475223 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn"] Jul 16 21:43:24.481236 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:24.481214 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n4wnn"] Jul 16 21:43:25.128367 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.128334 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w"] Jul 16 21:43:25.128682 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.128670 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" containerName="revoke-keys" Jul 16 21:43:25.128730 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.128685 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" containerName="revoke-keys" Jul 16 21:43:25.128730 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.128695 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" containerName="revoke-keys" Jul 16 21:43:25.128730 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.128700 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" containerName="revoke-keys" Jul 16 21:43:25.128730 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.128705 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eeac645-908f-4137-a50d-57a686d04f9e" containerName="revoke-keys" Jul 16 21:43:25.128730 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.128711 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eeac645-908f-4137-a50d-57a686d04f9e" containerName="revoke-keys" Jul 16 21:43:25.128730 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.128723 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eeac645-908f-4137-a50d-57a686d04f9e" containerName="revoke-keys" Jul 16 21:43:25.128730 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.128728 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eeac645-908f-4137-a50d-57a686d04f9e" containerName="revoke-keys" Jul 16 21:43:25.128940 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.128789 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eeac645-908f-4137-a50d-57a686d04f9e" containerName="revoke-keys" Jul 16 21:43:25.128940 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.128796 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eeac645-908f-4137-a50d-57a686d04f9e" containerName="revoke-keys" Jul 16 21:43:25.128940 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.128802 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" containerName="revoke-keys" Jul 16 21:43:25.128940 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.128808 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" containerName="revoke-keys" Jul 16 21:43:25.128940 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.128815 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" containerName="revoke-keys" Jul 16 21:43:25.128940 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.128820 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eeac645-908f-4137-a50d-57a686d04f9e" containerName="revoke-keys" Jul 16 21:43:25.132834 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.132817 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w" Jul 16 21:43:25.135928 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.135912 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:43:25.141412 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.141392 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w"] Jul 16 21:43:25.306670 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.306634 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d6d339-7dce-4b1a-9db0-65f49bcc114e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w\" (UID: \"d5d6d339-7dce-4b1a-9db0-65f49bcc114e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w" Jul 16 21:43:25.408108 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.408010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d6d339-7dce-4b1a-9db0-65f49bcc114e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w\" (UID: \"d5d6d339-7dce-4b1a-9db0-65f49bcc114e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w" Jul 16 21:43:25.408618 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.408594 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d6d339-7dce-4b1a-9db0-65f49bcc114e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w\" (UID: \"d5d6d339-7dce-4b1a-9db0-65f49bcc114e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w" Jul 16 21:43:25.443913 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.443878 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w" Jul 16 21:43:25.573042 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:25.573014 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w"] Jul 16 21:43:25.574256 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:43:25.574231 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d6d339_7dce_4b1a_9db0_65f49bcc114e.slice/crio-54e2bde9a3fbe0079bbdfa6fa19fcd2f21f00043fec9879b291deecfd0c2192a WatchSource:0}: Error finding container 54e2bde9a3fbe0079bbdfa6fa19fcd2f21f00043fec9879b291deecfd0c2192a: Status 404 returned error can't find the container with id 54e2bde9a3fbe0079bbdfa6fa19fcd2f21f00043fec9879b291deecfd0c2192a Jul 16 21:43:26.450776 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:26.450692 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eeac645-908f-4137-a50d-57a686d04f9e" path="/var/lib/kubelet/pods/9eeac645-908f-4137-a50d-57a686d04f9e/volumes" Jul 16 21:43:26.464002 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:26.463969 2577 generic.go:356] "Generic (PLEG): container finished" podID="d5d6d339-7dce-4b1a-9db0-65f49bcc114e" containerID="a06962edd297eca9dec06c06f5b633ed67bc2674a9b1105f714386dcd8f6d582" exitCode=6 Jul 16 21:43:26.464138 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:26.464053 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w" event={"ID":"d5d6d339-7dce-4b1a-9db0-65f49bcc114e","Type":"ContainerDied","Data":"a06962edd297eca9dec06c06f5b633ed67bc2674a9b1105f714386dcd8f6d582"} Jul 16 21:43:26.464138 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:26.464103 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w" event={"ID":"d5d6d339-7dce-4b1a-9db0-65f49bcc114e","Type":"ContainerStarted","Data":"54e2bde9a3fbe0079bbdfa6fa19fcd2f21f00043fec9879b291deecfd0c2192a"} Jul 16 21:43:26.464248 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:26.464206 2577 scope.go:117] "RemoveContainer" containerID="a06962edd297eca9dec06c06f5b633ed67bc2674a9b1105f714386dcd8f6d582" Jul 16 21:43:27.470482 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:27.470451 2577 generic.go:356] "Generic (PLEG): container finished" podID="d5d6d339-7dce-4b1a-9db0-65f49bcc114e" containerID="6c823b35d1e24eea329285a66b6e8e2cdc61770d2c9de5895041587648583a57" exitCode=6 Jul 16 21:43:27.470890 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:27.470504 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w" event={"ID":"d5d6d339-7dce-4b1a-9db0-65f49bcc114e","Type":"ContainerDied","Data":"6c823b35d1e24eea329285a66b6e8e2cdc61770d2c9de5895041587648583a57"} Jul 16 21:43:27.470890 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:27.470550 2577 scope.go:117] "RemoveContainer" containerID="a06962edd297eca9dec06c06f5b633ed67bc2674a9b1105f714386dcd8f6d582" Jul 16 21:43:27.470890 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:27.470708 2577 scope.go:117] "RemoveContainer" containerID="6c823b35d1e24eea329285a66b6e8e2cdc61770d2c9de5895041587648583a57" Jul 16 21:43:27.471034 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:27.470934 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w_odh-ai-gateway-infra(d5d6d339-7dce-4b1a-9db0-65f49bcc114e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w" podUID="d5d6d339-7dce-4b1a-9db0-65f49bcc114e" Jul 16 21:43:28.477086 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:28.477059 2577 scope.go:117] "RemoveContainer" containerID="6c823b35d1e24eea329285a66b6e8e2cdc61770d2c9de5895041587648583a57" Jul 16 21:43:28.477524 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:28.477289 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w_odh-ai-gateway-infra(d5d6d339-7dce-4b1a-9db0-65f49bcc114e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w" podUID="d5d6d339-7dce-4b1a-9db0-65f49bcc114e" Jul 16 21:43:36.444820 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:36.444785 2577 scope.go:117] "RemoveContainer" containerID="3da48d7c9a6ac95a50f22924eaa7c344386bd9dba06c700d039a2dc6e8e6db95" Jul 16 21:43:37.521529 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:37.521495 2577 generic.go:356] "Generic (PLEG): container finished" podID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerID="d5150007b0c84271b523700a9cd8beb66ca8450f2320180bc70c5a049215d710" exitCode=1 Jul 16 21:43:37.521948 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:37.521549 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" event={"ID":"454a4f80-c2dd-45b4-9b59-9152db8367d1","Type":"ContainerDied","Data":"d5150007b0c84271b523700a9cd8beb66ca8450f2320180bc70c5a049215d710"} Jul 16 21:43:37.521948 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:37.521594 2577 scope.go:117] "RemoveContainer" containerID="3da48d7c9a6ac95a50f22924eaa7c344386bd9dba06c700d039a2dc6e8e6db95" Jul 16 21:43:37.521948 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:37.521924 2577 scope.go:117] "RemoveContainer" containerID="d5150007b0c84271b523700a9cd8beb66ca8450f2320180bc70c5a049215d710" Jul 16 21:43:37.522116 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:37.522095 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:43:40.404045 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:40.404008 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:43:40.404465 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:40.404416 2577 scope.go:117] "RemoveContainer" containerID="d5150007b0c84271b523700a9cd8beb66ca8450f2320180bc70c5a049215d710" Jul 16 21:43:40.404610 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:40.404589 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:43:40.444714 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:40.444685 2577 scope.go:117] "RemoveContainer" containerID="6c823b35d1e24eea329285a66b6e8e2cdc61770d2c9de5895041587648583a57" Jul 16 21:43:41.544553 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:41.544516 2577 generic.go:356] "Generic (PLEG): container finished" podID="d5d6d339-7dce-4b1a-9db0-65f49bcc114e" containerID="f972b1689c3744ff6bf8359903d8125ab3abc58bdc563355cc5fa283d7cb94f7" exitCode=6 Jul 16 21:43:41.544953 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:41.544591 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w" event={"ID":"d5d6d339-7dce-4b1a-9db0-65f49bcc114e","Type":"ContainerDied","Data":"f972b1689c3744ff6bf8359903d8125ab3abc58bdc563355cc5fa283d7cb94f7"} Jul 16 21:43:41.544953 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:41.544653 2577 scope.go:117] "RemoveContainer" containerID="6c823b35d1e24eea329285a66b6e8e2cdc61770d2c9de5895041587648583a57" Jul 16 21:43:41.544953 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:41.544947 2577 scope.go:117] "RemoveContainer" containerID="f972b1689c3744ff6bf8359903d8125ab3abc58bdc563355cc5fa283d7cb94f7" Jul 16 21:43:41.545204 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:41.545165 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w_odh-ai-gateway-infra(d5d6d339-7dce-4b1a-9db0-65f49bcc114e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w" podUID="d5d6d339-7dce-4b1a-9db0-65f49bcc114e" Jul 16 21:43:42.081515 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:42.081485 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:43:42.081858 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:42.081845 2577 scope.go:117] "RemoveContainer" containerID="d5150007b0c84271b523700a9cd8beb66ca8450f2320180bc70c5a049215d710" Jul 16 21:43:42.082018 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:42.082000 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:43:42.584735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:42.584701 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w"] Jul 16 21:43:42.720916 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:42.720893 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w" Jul 16 21:43:42.761942 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:42.761910 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d6d339-7dce-4b1a-9db0-65f49bcc114e-maas-api-service-ca\") pod \"d5d6d339-7dce-4b1a-9db0-65f49bcc114e\" (UID: \"d5d6d339-7dce-4b1a-9db0-65f49bcc114e\") " Jul 16 21:43:42.762301 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:42.762276 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d6d339-7dce-4b1a-9db0-65f49bcc114e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d5d6d339-7dce-4b1a-9db0-65f49bcc114e" (UID: "d5d6d339-7dce-4b1a-9db0-65f49bcc114e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:43:42.863367 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:42.863282 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d6d339-7dce-4b1a-9db0-65f49bcc114e-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:43:43.557733 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:43.557703 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w" Jul 16 21:43:43.557900 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:43.557741 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w" event={"ID":"d5d6d339-7dce-4b1a-9db0-65f49bcc114e","Type":"ContainerDied","Data":"54e2bde9a3fbe0079bbdfa6fa19fcd2f21f00043fec9879b291deecfd0c2192a"} Jul 16 21:43:43.557900 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:43.557799 2577 scope.go:117] "RemoveContainer" containerID="f972b1689c3744ff6bf8359903d8125ab3abc58bdc563355cc5fa283d7cb94f7" Jul 16 21:43:43.585808 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:43.585778 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w"] Jul 16 21:43:43.590025 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:43.589998 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-qzj5w"] Jul 16 21:43:44.452278 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:44.452249 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5d6d339-7dce-4b1a-9db0-65f49bcc114e" path="/var/lib/kubelet/pods/d5d6d339-7dce-4b1a-9db0-65f49bcc114e/volumes" Jul 16 21:43:56.349309 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.349276 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn"] Jul 16 21:43:56.349679 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.349619 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d6d339-7dce-4b1a-9db0-65f49bcc114e" containerName="revoke-keys" Jul 16 21:43:56.349679 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.349629 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d6d339-7dce-4b1a-9db0-65f49bcc114e" containerName="revoke-keys" Jul 16 21:43:56.349679 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.349639 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" containerName="revoke-keys" Jul 16 21:43:56.349679 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.349645 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2dc4b8f-806a-4993-9a33-6f24e96c0ef0" containerName="revoke-keys" Jul 16 21:43:56.349679 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.349651 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eeac645-908f-4137-a50d-57a686d04f9e" containerName="revoke-keys" Jul 16 21:43:56.349679 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.349657 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eeac645-908f-4137-a50d-57a686d04f9e" containerName="revoke-keys" Jul 16 21:43:56.349679 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.349671 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d6d339-7dce-4b1a-9db0-65f49bcc114e" containerName="revoke-keys" Jul 16 21:43:56.349679 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.349677 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d6d339-7dce-4b1a-9db0-65f49bcc114e" containerName="revoke-keys" Jul 16 21:43:56.349929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.349685 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d6d339-7dce-4b1a-9db0-65f49bcc114e" containerName="revoke-keys" Jul 16 21:43:56.349929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.349689 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d6d339-7dce-4b1a-9db0-65f49bcc114e" containerName="revoke-keys" Jul 16 21:43:56.349929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.349744 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5d6d339-7dce-4b1a-9db0-65f49bcc114e" containerName="revoke-keys" Jul 16 21:43:56.349929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.349751 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5d6d339-7dce-4b1a-9db0-65f49bcc114e" containerName="revoke-keys" Jul 16 21:43:56.349929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.349757 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5d6d339-7dce-4b1a-9db0-65f49bcc114e" containerName="revoke-keys" Jul 16 21:43:56.352754 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.352735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" Jul 16 21:43:56.357563 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.357544 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:43:56.371595 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.371570 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn"] Jul 16 21:43:56.445827 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.445798 2577 scope.go:117] "RemoveContainer" containerID="d5150007b0c84271b523700a9cd8beb66ca8450f2320180bc70c5a049215d710" Jul 16 21:43:56.446051 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:56.446028 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:43:56.482020 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.481992 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c7ac1d-8f5e-411a-aa44-bd6243710afc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn\" (UID: \"86c7ac1d-8f5e-411a-aa44-bd6243710afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" Jul 16 21:43:56.582887 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.582847 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c7ac1d-8f5e-411a-aa44-bd6243710afc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn\" (UID: \"86c7ac1d-8f5e-411a-aa44-bd6243710afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" Jul 16 21:43:56.583501 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.583479 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c7ac1d-8f5e-411a-aa44-bd6243710afc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn\" (UID: \"86c7ac1d-8f5e-411a-aa44-bd6243710afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" Jul 16 21:43:56.662374 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:56.662347 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" Jul 16 21:43:57.008617 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:57.008587 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn"] Jul 16 21:43:57.009622 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:43:57.009593 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c7ac1d_8f5e_411a_aa44_bd6243710afc.slice/crio-33119c74b5332e47fbc9522bf0965de5278570be69d8f8e1990e36846bfa194c WatchSource:0}: Error finding container 33119c74b5332e47fbc9522bf0965de5278570be69d8f8e1990e36846bfa194c: Status 404 returned error can't find the container with id 33119c74b5332e47fbc9522bf0965de5278570be69d8f8e1990e36846bfa194c Jul 16 21:43:57.626969 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:57.626943 2577 generic.go:356] "Generic (PLEG): container finished" podID="86c7ac1d-8f5e-411a-aa44-bd6243710afc" containerID="e8d38f55573d8f0167d39b011816b8bf36f14b3cf8593eb7d0bfabc4301ec51a" exitCode=6 Jul 16 21:43:57.627336 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:57.627008 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" event={"ID":"86c7ac1d-8f5e-411a-aa44-bd6243710afc","Type":"ContainerDied","Data":"e8d38f55573d8f0167d39b011816b8bf36f14b3cf8593eb7d0bfabc4301ec51a"} Jul 16 21:43:57.627336 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:57.627059 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" event={"ID":"86c7ac1d-8f5e-411a-aa44-bd6243710afc","Type":"ContainerStarted","Data":"33119c74b5332e47fbc9522bf0965de5278570be69d8f8e1990e36846bfa194c"} Jul 16 21:43:57.627336 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:57.627243 2577 scope.go:117] "RemoveContainer" containerID="e8d38f55573d8f0167d39b011816b8bf36f14b3cf8593eb7d0bfabc4301ec51a" Jul 16 21:43:58.634056 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:58.634025 2577 generic.go:356] "Generic (PLEG): container finished" podID="86c7ac1d-8f5e-411a-aa44-bd6243710afc" containerID="25be1bd55c63fe21f6518ffd365d06e963cbd99ff85f8524ef390af8607790ee" exitCode=6 Jul 16 21:43:58.634482 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:58.634111 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" event={"ID":"86c7ac1d-8f5e-411a-aa44-bd6243710afc","Type":"ContainerDied","Data":"25be1bd55c63fe21f6518ffd365d06e963cbd99ff85f8524ef390af8607790ee"} Jul 16 21:43:58.634482 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:58.634173 2577 scope.go:117] "RemoveContainer" containerID="e8d38f55573d8f0167d39b011816b8bf36f14b3cf8593eb7d0bfabc4301ec51a" Jul 16 21:43:58.634482 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:58.634310 2577 scope.go:117] "RemoveContainer" containerID="25be1bd55c63fe21f6518ffd365d06e963cbd99ff85f8524ef390af8607790ee" Jul 16 21:43:58.634667 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:58.634598 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn_odh-ai-gateway-infra(86c7ac1d-8f5e-411a-aa44-bd6243710afc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" podUID="86c7ac1d-8f5e-411a-aa44-bd6243710afc" Jul 16 21:43:59.640896 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:43:59.640870 2577 scope.go:117] "RemoveContainer" containerID="25be1bd55c63fe21f6518ffd365d06e963cbd99ff85f8524ef390af8607790ee" Jul 16 21:43:59.641356 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:43:59.641112 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn_odh-ai-gateway-infra(86c7ac1d-8f5e-411a-aa44-bd6243710afc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" podUID="86c7ac1d-8f5e-411a-aa44-bd6243710afc" Jul 16 21:44:04.567752 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:04.567714 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w"] Jul 16 21:44:04.572383 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:04.572365 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" Jul 16 21:44:04.590780 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:04.590752 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w"] Jul 16 21:44:04.649750 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:04.649719 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a5e4f6a-7263-497e-aa85-352276f7f262-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w\" (UID: \"9a5e4f6a-7263-497e-aa85-352276f7f262\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" Jul 16 21:44:04.751176 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:04.751144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a5e4f6a-7263-497e-aa85-352276f7f262-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w\" (UID: \"9a5e4f6a-7263-497e-aa85-352276f7f262\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" Jul 16 21:44:04.751756 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:04.751734 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a5e4f6a-7263-497e-aa85-352276f7f262-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w\" (UID: \"9a5e4f6a-7263-497e-aa85-352276f7f262\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" Jul 16 21:44:04.882350 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:04.882320 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" Jul 16 21:44:05.255686 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:05.255660 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w"] Jul 16 21:44:05.257561 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:44:05.257537 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5e4f6a_7263_497e_aa85_352276f7f262.slice/crio-5ca51c9545cc4fc7e00322d3e44dcdd67a3359e47cad9e34de3026fb902c2d19 WatchSource:0}: Error finding container 5ca51c9545cc4fc7e00322d3e44dcdd67a3359e47cad9e34de3026fb902c2d19: Status 404 returned error can't find the container with id 5ca51c9545cc4fc7e00322d3e44dcdd67a3359e47cad9e34de3026fb902c2d19 Jul 16 21:44:05.673300 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:05.673257 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" event={"ID":"9a5e4f6a-7263-497e-aa85-352276f7f262","Type":"ContainerStarted","Data":"4958f9363c9b6b4445bff3ab15fa3a96d3ca63a9e21b7c1d5045cc6b83ebedc1"} Jul 16 21:44:05.673732 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:05.673315 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" event={"ID":"9a5e4f6a-7263-497e-aa85-352276f7f262","Type":"ContainerStarted","Data":"5ca51c9545cc4fc7e00322d3e44dcdd67a3359e47cad9e34de3026fb902c2d19"} Jul 16 21:44:06.679952 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:06.679917 2577 generic.go:356] "Generic (PLEG): container finished" podID="9a5e4f6a-7263-497e-aa85-352276f7f262" containerID="4958f9363c9b6b4445bff3ab15fa3a96d3ca63a9e21b7c1d5045cc6b83ebedc1" exitCode=7 Jul 16 21:44:06.680416 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:06.679973 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" event={"ID":"9a5e4f6a-7263-497e-aa85-352276f7f262","Type":"ContainerDied","Data":"4958f9363c9b6b4445bff3ab15fa3a96d3ca63a9e21b7c1d5045cc6b83ebedc1"} Jul 16 21:44:06.680416 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:06.680201 2577 scope.go:117] "RemoveContainer" containerID="4958f9363c9b6b4445bff3ab15fa3a96d3ca63a9e21b7c1d5045cc6b83ebedc1" Jul 16 21:44:07.686895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:07.686857 2577 generic.go:356] "Generic (PLEG): container finished" podID="9a5e4f6a-7263-497e-aa85-352276f7f262" containerID="91cd227d335e82af2c4c9903c719020659847d3a3fcc84cf1da00e964f9eb526" exitCode=7 Jul 16 21:44:07.687352 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:07.686932 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" event={"ID":"9a5e4f6a-7263-497e-aa85-352276f7f262","Type":"ContainerDied","Data":"91cd227d335e82af2c4c9903c719020659847d3a3fcc84cf1da00e964f9eb526"} Jul 16 21:44:07.687352 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:07.686980 2577 scope.go:117] "RemoveContainer" containerID="4958f9363c9b6b4445bff3ab15fa3a96d3ca63a9e21b7c1d5045cc6b83ebedc1" Jul 16 21:44:07.687352 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:07.687152 2577 scope.go:117] "RemoveContainer" containerID="91cd227d335e82af2c4c9903c719020659847d3a3fcc84cf1da00e964f9eb526" Jul 16 21:44:07.687463 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:44:07.687400 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w_odh-ai-gateway-infra(9a5e4f6a-7263-497e-aa85-352276f7f262)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" podUID="9a5e4f6a-7263-497e-aa85-352276f7f262" Jul 16 21:44:08.693599 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:08.693571 2577 scope.go:117] "RemoveContainer" containerID="91cd227d335e82af2c4c9903c719020659847d3a3fcc84cf1da00e964f9eb526" Jul 16 21:44:08.693981 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:44:08.693740 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w_odh-ai-gateway-infra(9a5e4f6a-7263-497e-aa85-352276f7f262)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" podUID="9a5e4f6a-7263-497e-aa85-352276f7f262" Jul 16 21:44:09.445386 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:09.445353 2577 scope.go:117] "RemoveContainer" containerID="d5150007b0c84271b523700a9cd8beb66ca8450f2320180bc70c5a049215d710" Jul 16 21:44:09.445567 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:44:09.445541 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:44:12.445343 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:12.445306 2577 scope.go:117] "RemoveContainer" containerID="25be1bd55c63fe21f6518ffd365d06e963cbd99ff85f8524ef390af8607790ee" Jul 16 21:44:12.717644 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:12.717563 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" event={"ID":"86c7ac1d-8f5e-411a-aa44-bd6243710afc","Type":"ContainerStarted","Data":"5da8af39471027e9645ddf7374f78d8085e4dd8cc3ae3d7bc350de657df625f4"} Jul 16 21:44:12.752139 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:12.752085 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" podStartSLOduration=16.752068025 podStartE2EDuration="16.752068025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:44:12.743626037 +0000 UTC m=+2068.918734229" watchObservedRunningTime="2026-07-16 21:44:12.752068025 +0000 UTC m=+2068.927176216" Jul 16 21:44:13.724849 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:13.724815 2577 generic.go:356] "Generic (PLEG): container finished" podID="86c7ac1d-8f5e-411a-aa44-bd6243710afc" containerID="5da8af39471027e9645ddf7374f78d8085e4dd8cc3ae3d7bc350de657df625f4" exitCode=6 Jul 16 21:44:13.725272 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:13.724891 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" event={"ID":"86c7ac1d-8f5e-411a-aa44-bd6243710afc","Type":"ContainerDied","Data":"5da8af39471027e9645ddf7374f78d8085e4dd8cc3ae3d7bc350de657df625f4"} Jul 16 21:44:13.725272 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:13.724955 2577 scope.go:117] "RemoveContainer" containerID="25be1bd55c63fe21f6518ffd365d06e963cbd99ff85f8524ef390af8607790ee" Jul 16 21:44:13.725272 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:13.725209 2577 scope.go:117] "RemoveContainer" containerID="5da8af39471027e9645ddf7374f78d8085e4dd8cc3ae3d7bc350de657df625f4" Jul 16 21:44:13.725474 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:44:13.725435 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn_odh-ai-gateway-infra(86c7ac1d-8f5e-411a-aa44-bd6243710afc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" podUID="86c7ac1d-8f5e-411a-aa44-bd6243710afc" Jul 16 21:44:13.769533 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:13.769496 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn"] Jul 16 21:44:14.861616 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:14.861593 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" Jul 16 21:44:15.042897 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.042789 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c7ac1d-8f5e-411a-aa44-bd6243710afc-maas-api-service-ca\") pod \"86c7ac1d-8f5e-411a-aa44-bd6243710afc\" (UID: \"86c7ac1d-8f5e-411a-aa44-bd6243710afc\") " Jul 16 21:44:15.043155 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.043130 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c7ac1d-8f5e-411a-aa44-bd6243710afc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86c7ac1d-8f5e-411a-aa44-bd6243710afc" (UID: "86c7ac1d-8f5e-411a-aa44-bd6243710afc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:44:15.143658 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.143610 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c7ac1d-8f5e-411a-aa44-bd6243710afc-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:44:15.216487 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.216451 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p"] Jul 16 21:44:15.216829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.216817 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c7ac1d-8f5e-411a-aa44-bd6243710afc" containerName="revoke-keys" Jul 16 21:44:15.216873 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.216830 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c7ac1d-8f5e-411a-aa44-bd6243710afc" containerName="revoke-keys" Jul 16 21:44:15.216873 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.216852 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c7ac1d-8f5e-411a-aa44-bd6243710afc" containerName="revoke-keys" Jul 16 21:44:15.216873 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.216858 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c7ac1d-8f5e-411a-aa44-bd6243710afc" containerName="revoke-keys" Jul 16 21:44:15.216970 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.216915 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c7ac1d-8f5e-411a-aa44-bd6243710afc" containerName="revoke-keys" Jul 16 21:44:15.216970 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.216924 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c7ac1d-8f5e-411a-aa44-bd6243710afc" containerName="revoke-keys" Jul 16 21:44:15.219980 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.219963 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p" Jul 16 21:44:15.241498 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.241470 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p"] Jul 16 21:44:15.345344 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.345259 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fc72fe-0c98-44f1-b6bf-f3cc59a453ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p\" (UID: \"94fc72fe-0c98-44f1-b6bf-f3cc59a453ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p" Jul 16 21:44:15.446272 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.446240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fc72fe-0c98-44f1-b6bf-f3cc59a453ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p\" (UID: \"94fc72fe-0c98-44f1-b6bf-f3cc59a453ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p" Jul 16 21:44:15.446878 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.446851 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fc72fe-0c98-44f1-b6bf-f3cc59a453ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p\" (UID: \"94fc72fe-0c98-44f1-b6bf-f3cc59a453ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p" Jul 16 21:44:15.529452 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.529416 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p" Jul 16 21:44:15.688168 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.688141 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p"] Jul 16 21:44:15.689275 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:44:15.689247 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fc72fe_0c98_44f1_b6bf_f3cc59a453ad.slice/crio-033a677dd7f3cb6745e629a584e9a03d682136b3187072fe81ffd6614dd3eca7 WatchSource:0}: Error finding container 033a677dd7f3cb6745e629a584e9a03d682136b3187072fe81ffd6614dd3eca7: Status 404 returned error can't find the container with id 033a677dd7f3cb6745e629a584e9a03d682136b3187072fe81ffd6614dd3eca7 Jul 16 21:44:15.738326 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.738302 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" Jul 16 21:44:15.738326 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.738305 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn" event={"ID":"86c7ac1d-8f5e-411a-aa44-bd6243710afc","Type":"ContainerDied","Data":"33119c74b5332e47fbc9522bf0965de5278570be69d8f8e1990e36846bfa194c"} Jul 16 21:44:15.738514 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.738372 2577 scope.go:117] "RemoveContainer" containerID="5da8af39471027e9645ddf7374f78d8085e4dd8cc3ae3d7bc350de657df625f4" Jul 16 21:44:15.739627 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.739603 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p" event={"ID":"94fc72fe-0c98-44f1-b6bf-f3cc59a453ad","Type":"ContainerStarted","Data":"033a677dd7f3cb6745e629a584e9a03d682136b3187072fe81ffd6614dd3eca7"} Jul 16 21:44:15.779227 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.779179 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn"] Jul 16 21:44:15.788373 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:15.788350 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vhzqn"] Jul 16 21:44:16.451032 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:16.450991 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86c7ac1d-8f5e-411a-aa44-bd6243710afc" path="/var/lib/kubelet/pods/86c7ac1d-8f5e-411a-aa44-bd6243710afc/volumes" Jul 16 21:44:16.747218 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:16.747116 2577 generic.go:356] "Generic (PLEG): container finished" podID="94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" containerID="612200236debbcb44361a02be037cb5e52e354ad501370cc01599646f066eb84" exitCode=6 Jul 16 21:44:16.747218 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:16.747152 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p" event={"ID":"94fc72fe-0c98-44f1-b6bf-f3cc59a453ad","Type":"ContainerDied","Data":"612200236debbcb44361a02be037cb5e52e354ad501370cc01599646f066eb84"} Jul 16 21:44:16.747430 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:16.747415 2577 scope.go:117] "RemoveContainer" containerID="612200236debbcb44361a02be037cb5e52e354ad501370cc01599646f066eb84" Jul 16 21:44:17.753980 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:17.753896 2577 generic.go:356] "Generic (PLEG): container finished" podID="94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" containerID="12dd9240c6aac5a6703f02a2bd7d27576ce8524c66adadf01904571ed939776d" exitCode=6 Jul 16 21:44:17.754396 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:17.753979 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p" event={"ID":"94fc72fe-0c98-44f1-b6bf-f3cc59a453ad","Type":"ContainerDied","Data":"12dd9240c6aac5a6703f02a2bd7d27576ce8524c66adadf01904571ed939776d"} Jul 16 21:44:17.754396 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:17.754064 2577 scope.go:117] "RemoveContainer" containerID="612200236debbcb44361a02be037cb5e52e354ad501370cc01599646f066eb84" Jul 16 21:44:17.754396 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:17.754307 2577 scope.go:117] "RemoveContainer" containerID="12dd9240c6aac5a6703f02a2bd7d27576ce8524c66adadf01904571ed939776d" Jul 16 21:44:17.754542 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:44:17.754522 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p_odh-ai-gateway-infra(94fc72fe-0c98-44f1-b6bf-f3cc59a453ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p" podUID="94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" Jul 16 21:44:18.760773 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:18.760741 2577 scope.go:117] "RemoveContainer" containerID="12dd9240c6aac5a6703f02a2bd7d27576ce8524c66adadf01904571ed939776d" Jul 16 21:44:18.761149 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:44:18.760945 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p_odh-ai-gateway-infra(94fc72fe-0c98-44f1-b6bf-f3cc59a453ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p" podUID="94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" Jul 16 21:44:20.450318 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:20.450291 2577 scope.go:117] "RemoveContainer" containerID="d5150007b0c84271b523700a9cd8beb66ca8450f2320180bc70c5a049215d710" Jul 16 21:44:20.450664 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:44:20.450457 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:44:23.444953 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:23.444920 2577 scope.go:117] "RemoveContainer" containerID="91cd227d335e82af2c4c9903c719020659847d3a3fcc84cf1da00e964f9eb526" Jul 16 21:44:23.787699 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:23.787621 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" event={"ID":"9a5e4f6a-7263-497e-aa85-352276f7f262","Type":"ContainerStarted","Data":"977de52134fa4552d879550e872ca6b6186dd1987a3d6b3dd9c38812f2c98382"} Jul 16 21:44:23.818658 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:23.818606 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" podStartSLOduration=19.818591107 podStartE2EDuration="19.818591107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:44:05.707769103 +0000 UTC m=+2061.882877295" watchObservedRunningTime="2026-07-16 21:44:23.818591107 +0000 UTC m=+2079.993699299" Jul 16 21:44:24.794851 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:24.794819 2577 generic.go:356] "Generic (PLEG): container finished" podID="9a5e4f6a-7263-497e-aa85-352276f7f262" containerID="977de52134fa4552d879550e872ca6b6186dd1987a3d6b3dd9c38812f2c98382" exitCode=7 Jul 16 21:44:24.795351 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:24.794877 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" event={"ID":"9a5e4f6a-7263-497e-aa85-352276f7f262","Type":"ContainerDied","Data":"977de52134fa4552d879550e872ca6b6186dd1987a3d6b3dd9c38812f2c98382"} Jul 16 21:44:24.795351 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:24.794935 2577 scope.go:117] "RemoveContainer" containerID="91cd227d335e82af2c4c9903c719020659847d3a3fcc84cf1da00e964f9eb526" Jul 16 21:44:24.795351 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:24.795227 2577 scope.go:117] "RemoveContainer" containerID="977de52134fa4552d879550e872ca6b6186dd1987a3d6b3dd9c38812f2c98382" Jul 16 21:44:24.795473 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:44:24.795426 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w_odh-ai-gateway-infra(9a5e4f6a-7263-497e-aa85-352276f7f262)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" podUID="9a5e4f6a-7263-497e-aa85-352276f7f262" Jul 16 21:44:24.843788 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:24.843758 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w"] Jul 16 21:44:25.959279 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:25.959249 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" Jul 16 21:44:26.041533 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:26.041495 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a5e4f6a-7263-497e-aa85-352276f7f262-maas-api-service-ca\") pod \"9a5e4f6a-7263-497e-aa85-352276f7f262\" (UID: \"9a5e4f6a-7263-497e-aa85-352276f7f262\") " Jul 16 21:44:26.041832 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:26.041810 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a5e4f6a-7263-497e-aa85-352276f7f262-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a5e4f6a-7263-497e-aa85-352276f7f262" (UID: "9a5e4f6a-7263-497e-aa85-352276f7f262"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:44:26.142755 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:26.142727 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a5e4f6a-7263-497e-aa85-352276f7f262-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:44:26.814385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:26.814294 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" Jul 16 21:44:26.814385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:26.814329 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w" event={"ID":"9a5e4f6a-7263-497e-aa85-352276f7f262","Type":"ContainerDied","Data":"5ca51c9545cc4fc7e00322d3e44dcdd67a3359e47cad9e34de3026fb902c2d19"} Jul 16 21:44:26.814663 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:26.814417 2577 scope.go:117] "RemoveContainer" containerID="977de52134fa4552d879550e872ca6b6186dd1987a3d6b3dd9c38812f2c98382" Jul 16 21:44:26.844734 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:26.844695 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w"] Jul 16 21:44:26.855840 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:26.855805 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-t5q9w"] Jul 16 21:44:28.450141 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:28.450106 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a5e4f6a-7263-497e-aa85-352276f7f262" path="/var/lib/kubelet/pods/9a5e4f6a-7263-497e-aa85-352276f7f262/volumes" Jul 16 21:44:32.451242 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:32.451215 2577 scope.go:117] "RemoveContainer" containerID="d5150007b0c84271b523700a9cd8beb66ca8450f2320180bc70c5a049215d710" Jul 16 21:44:32.451644 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:32.451333 2577 scope.go:117] "RemoveContainer" containerID="12dd9240c6aac5a6703f02a2bd7d27576ce8524c66adadf01904571ed939776d" Jul 16 21:44:32.451644 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:44:32.451408 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:44:32.848490 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:32.848453 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p" event={"ID":"94fc72fe-0c98-44f1-b6bf-f3cc59a453ad","Type":"ContainerStarted","Data":"60ee93209802c826f5139d08bdbef691b8def71a7ec4bceeec5b24a04a5b490c"} Jul 16 21:44:33.855123 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:33.855089 2577 generic.go:356] "Generic (PLEG): container finished" podID="94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" containerID="60ee93209802c826f5139d08bdbef691b8def71a7ec4bceeec5b24a04a5b490c" exitCode=6 Jul 16 21:44:33.855570 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:33.855165 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p" event={"ID":"94fc72fe-0c98-44f1-b6bf-f3cc59a453ad","Type":"ContainerDied","Data":"60ee93209802c826f5139d08bdbef691b8def71a7ec4bceeec5b24a04a5b490c"} Jul 16 21:44:33.855570 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:33.855243 2577 scope.go:117] "RemoveContainer" containerID="12dd9240c6aac5a6703f02a2bd7d27576ce8524c66adadf01904571ed939776d" Jul 16 21:44:33.855570 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:33.855544 2577 scope.go:117] "RemoveContainer" containerID="60ee93209802c826f5139d08bdbef691b8def71a7ec4bceeec5b24a04a5b490c" Jul 16 21:44:33.855759 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:44:33.855740 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p_odh-ai-gateway-infra(94fc72fe-0c98-44f1-b6bf-f3cc59a453ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p" podUID="94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" Jul 16 21:44:33.931710 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:33.931674 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p"] Jul 16 21:44:34.992428 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:34.992401 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p" Jul 16 21:44:35.021632 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:35.021602 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fc72fe-0c98-44f1-b6bf-f3cc59a453ad-maas-api-service-ca\") pod \"94fc72fe-0c98-44f1-b6bf-f3cc59a453ad\" (UID: \"94fc72fe-0c98-44f1-b6bf-f3cc59a453ad\") " Jul 16 21:44:35.021927 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:35.021907 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94fc72fe-0c98-44f1-b6bf-f3cc59a453ad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" (UID: "94fc72fe-0c98-44f1-b6bf-f3cc59a453ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:44:35.122631 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:35.122550 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fc72fe-0c98-44f1-b6bf-f3cc59a453ad-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:44:35.869327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:35.869292 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p" event={"ID":"94fc72fe-0c98-44f1-b6bf-f3cc59a453ad","Type":"ContainerDied","Data":"033a677dd7f3cb6745e629a584e9a03d682136b3187072fe81ffd6614dd3eca7"} Jul 16 21:44:35.869327 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:35.869327 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p" Jul 16 21:44:35.869587 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:35.869365 2577 scope.go:117] "RemoveContainer" containerID="60ee93209802c826f5139d08bdbef691b8def71a7ec4bceeec5b24a04a5b490c" Jul 16 21:44:35.905382 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:35.905353 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p"] Jul 16 21:44:35.912497 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:35.912473 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fgc9p"] Jul 16 21:44:36.450787 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:36.450743 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" path="/var/lib/kubelet/pods/94fc72fe-0c98-44f1-b6bf-f3cc59a453ad/volumes" Jul 16 21:44:44.438513 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:44.438484 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:44:44.442472 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:44.442450 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:44:44.448128 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:44.448102 2577 scope.go:117] "RemoveContainer" containerID="d5150007b0c84271b523700a9cd8beb66ca8450f2320180bc70c5a049215d710" Jul 16 21:44:44.448622 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:44:44.448588 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:44:46.429803 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.429744 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v"] Jul 16 21:44:46.430317 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430289 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5e4f6a-7263-497e-aa85-352276f7f262" containerName="revoke-keys" Jul 16 21:44:46.430404 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430319 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5e4f6a-7263-497e-aa85-352276f7f262" containerName="revoke-keys" Jul 16 21:44:46.430404 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430339 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5e4f6a-7263-497e-aa85-352276f7f262" containerName="revoke-keys" Jul 16 21:44:46.430404 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430347 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5e4f6a-7263-497e-aa85-352276f7f262" containerName="revoke-keys" Jul 16 21:44:46.430404 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430358 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5e4f6a-7263-497e-aa85-352276f7f262" containerName="revoke-keys" Jul 16 21:44:46.430404 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430367 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5e4f6a-7263-497e-aa85-352276f7f262" containerName="revoke-keys" Jul 16 21:44:46.430404 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430380 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c7ac1d-8f5e-411a-aa44-bd6243710afc" containerName="revoke-keys" Jul 16 21:44:46.430404 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430390 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c7ac1d-8f5e-411a-aa44-bd6243710afc" containerName="revoke-keys" Jul 16 21:44:46.430404 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430398 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" containerName="revoke-keys" Jul 16 21:44:46.430798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430409 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" containerName="revoke-keys" Jul 16 21:44:46.430798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430423 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" containerName="revoke-keys" Jul 16 21:44:46.430798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430432 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" containerName="revoke-keys" Jul 16 21:44:46.430798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430444 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" containerName="revoke-keys" Jul 16 21:44:46.430798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430453 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" containerName="revoke-keys" Jul 16 21:44:46.430798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430541 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c7ac1d-8f5e-411a-aa44-bd6243710afc" containerName="revoke-keys" Jul 16 21:44:46.430798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430575 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" containerName="revoke-keys" Jul 16 21:44:46.430798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430589 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5e4f6a-7263-497e-aa85-352276f7f262" containerName="revoke-keys" Jul 16 21:44:46.430798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430598 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5e4f6a-7263-497e-aa85-352276f7f262" containerName="revoke-keys" Jul 16 21:44:46.430798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430607 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" containerName="revoke-keys" Jul 16 21:44:46.430798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.430616 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5e4f6a-7263-497e-aa85-352276f7f262" containerName="revoke-keys" Jul 16 21:44:46.435624 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.435604 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v" Jul 16 21:44:46.439856 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.439831 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:44:46.484673 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.484636 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v"] Jul 16 21:44:46.524339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.524303 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda983e2-daa2-46c5-89c9-02a367dfcf67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v\" (UID: \"fda983e2-daa2-46c5-89c9-02a367dfcf67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v" Jul 16 21:44:46.625877 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.625841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda983e2-daa2-46c5-89c9-02a367dfcf67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v\" (UID: \"fda983e2-daa2-46c5-89c9-02a367dfcf67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v" Jul 16 21:44:46.626518 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.626495 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda983e2-daa2-46c5-89c9-02a367dfcf67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v\" (UID: \"fda983e2-daa2-46c5-89c9-02a367dfcf67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v" Jul 16 21:44:46.745868 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.745781 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v" Jul 16 21:44:46.907308 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.907278 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v"] Jul 16 21:44:46.909166 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:44:46.909137 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda983e2_daa2_46c5_89c9_02a367dfcf67.slice/crio-8b7cf571619536d5247e805f667a7114c22274fbb405859bdf56c91d3f2c392d WatchSource:0}: Error finding container 8b7cf571619536d5247e805f667a7114c22274fbb405859bdf56c91d3f2c392d: Status 404 returned error can't find the container with id 8b7cf571619536d5247e805f667a7114c22274fbb405859bdf56c91d3f2c392d Jul 16 21:44:46.931638 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:46.931606 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v" event={"ID":"fda983e2-daa2-46c5-89c9-02a367dfcf67","Type":"ContainerStarted","Data":"8b7cf571619536d5247e805f667a7114c22274fbb405859bdf56c91d3f2c392d"} Jul 16 21:44:47.939136 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:47.939100 2577 generic.go:356] "Generic (PLEG): container finished" podID="fda983e2-daa2-46c5-89c9-02a367dfcf67" containerID="ee45cf615d86663cf3e994ef84cfa57c1bcd74fe6961b93ce0b0527e327703cf" exitCode=6 Jul 16 21:44:47.939524 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:47.939162 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v" event={"ID":"fda983e2-daa2-46c5-89c9-02a367dfcf67","Type":"ContainerDied","Data":"ee45cf615d86663cf3e994ef84cfa57c1bcd74fe6961b93ce0b0527e327703cf"} Jul 16 21:44:47.939524 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:47.939445 2577 scope.go:117] "RemoveContainer" containerID="ee45cf615d86663cf3e994ef84cfa57c1bcd74fe6961b93ce0b0527e327703cf" Jul 16 21:44:48.945936 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:48.945896 2577 generic.go:356] "Generic (PLEG): container finished" podID="fda983e2-daa2-46c5-89c9-02a367dfcf67" containerID="dbcbd4b4f02ae7d04f4745e153d503e7a3825fcdb6aa9ac14992b5ec6b599d91" exitCode=6 Jul 16 21:44:48.946389 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:48.945943 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v" event={"ID":"fda983e2-daa2-46c5-89c9-02a367dfcf67","Type":"ContainerDied","Data":"dbcbd4b4f02ae7d04f4745e153d503e7a3825fcdb6aa9ac14992b5ec6b599d91"} Jul 16 21:44:48.946389 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:48.946005 2577 scope.go:117] "RemoveContainer" containerID="ee45cf615d86663cf3e994ef84cfa57c1bcd74fe6961b93ce0b0527e327703cf" Jul 16 21:44:48.946389 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:48.946221 2577 scope.go:117] "RemoveContainer" containerID="dbcbd4b4f02ae7d04f4745e153d503e7a3825fcdb6aa9ac14992b5ec6b599d91" Jul 16 21:44:48.946629 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:44:48.946441 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v_odh-ai-gateway-infra(fda983e2-daa2-46c5-89c9-02a367dfcf67)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v" podUID="fda983e2-daa2-46c5-89c9-02a367dfcf67" Jul 16 21:44:49.952874 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:49.952841 2577 scope.go:117] "RemoveContainer" containerID="dbcbd4b4f02ae7d04f4745e153d503e7a3825fcdb6aa9ac14992b5ec6b599d91" Jul 16 21:44:49.953374 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:44:49.953111 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v_odh-ai-gateway-infra(fda983e2-daa2-46c5-89c9-02a367dfcf67)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v" podUID="fda983e2-daa2-46c5-89c9-02a367dfcf67" Jul 16 21:44:59.445336 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:44:59.445297 2577 scope.go:117] "RemoveContainer" containerID="d5150007b0c84271b523700a9cd8beb66ca8450f2320180bc70c5a049215d710" Jul 16 21:45:00.005474 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:00.005441 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" event={"ID":"454a4f80-c2dd-45b4-9b59-9152db8367d1","Type":"ContainerStarted","Data":"fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249"} Jul 16 21:45:00.005660 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:00.005641 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:45:00.182037 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:00.182001 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737305-m79dl"] Jul 16 21:45:00.182463 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:00.182448 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="94fc72fe-0c98-44f1-b6bf-f3cc59a453ad" containerName="revoke-keys" Jul 16 21:45:00.185546 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:00.185530 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737305-m79dl" Jul 16 21:45:00.189021 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:00.188997 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg"] Jul 16 21:45:00.194302 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:00.194275 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg" Jul 16 21:45:00.203291 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:00.203266 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737305-m79dl" Jul 16 21:45:00.210745 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:00.210721 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg" Jul 16 21:45:00.212457 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:00.212432 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737305-m79dl"] Jul 16 21:45:00.220715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:00.218271 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg"] Jul 16 21:45:00.391019 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:00.390983 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg"] Jul 16 21:45:00.392070 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:45:00.392046 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89274d00_061a_4e73_9779_2bc6ffd22aa3.slice/crio-171bbeff4ce7be72cb9ba48e57caf6d1f2b31c5bffe38c91b2664144efdc3370 WatchSource:0}: Error finding container 171bbeff4ce7be72cb9ba48e57caf6d1f2b31c5bffe38c91b2664144efdc3370: Status 404 returned error can't find the container with id 171bbeff4ce7be72cb9ba48e57caf6d1f2b31c5bffe38c91b2664144efdc3370 Jul 16 21:45:00.445512 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:00.445483 2577 scope.go:117] "RemoveContainer" containerID="dbcbd4b4f02ae7d04f4745e153d503e7a3825fcdb6aa9ac14992b5ec6b599d91" Jul 16 21:45:00.590404 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:00.590377 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737305-m79dl"] Jul 16 21:45:00.593364 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:45:00.593336 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b80877d_921d_42d8_a8fa_74d0826123c3.slice/crio-1df5e0651d160446aa86c103f89876166fcf44a4a9cbd7e50db94e0b06054763 WatchSource:0}: Error finding container 1df5e0651d160446aa86c103f89876166fcf44a4a9cbd7e50db94e0b06054763: Status 404 returned error can't find the container with id 1df5e0651d160446aa86c103f89876166fcf44a4a9cbd7e50db94e0b06054763 Jul 16 21:45:01.011812 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:01.011776 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737305-m79dl" event={"ID":"3b80877d-921d-42d8-a8fa-74d0826123c3","Type":"ContainerStarted","Data":"c3decf7f6b29da77fbf1702d5fdbc00a45a91e6f8736a4d8ab5125f6e992daf7"} Jul 16 21:45:01.011958 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:01.011838 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737305-m79dl" event={"ID":"3b80877d-921d-42d8-a8fa-74d0826123c3","Type":"ContainerStarted","Data":"1df5e0651d160446aa86c103f89876166fcf44a4a9cbd7e50db94e0b06054763"} Jul 16 21:45:01.013350 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:01.013324 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg" event={"ID":"89274d00-061a-4e73-9779-2bc6ffd22aa3","Type":"ContainerStarted","Data":"c3d56d00838da16874c8994bd0f74db92bc95274692519b070dad2af362a4ec5"} Jul 16 21:45:01.013482 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:01.013370 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg" event={"ID":"89274d00-061a-4e73-9779-2bc6ffd22aa3","Type":"ContainerStarted","Data":"171bbeff4ce7be72cb9ba48e57caf6d1f2b31c5bffe38c91b2664144efdc3370"} Jul 16 21:45:01.014971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:01.014949 2577 generic.go:356] "Generic (PLEG): container finished" podID="fda983e2-daa2-46c5-89c9-02a367dfcf67" containerID="baa4c46659374a860096bbc0dcf15542c63011509e0f736190f9e517f86b847c" exitCode=6 Jul 16 21:45:01.015088 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:01.015023 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v" event={"ID":"fda983e2-daa2-46c5-89c9-02a367dfcf67","Type":"ContainerDied","Data":"baa4c46659374a860096bbc0dcf15542c63011509e0f736190f9e517f86b847c"} Jul 16 21:45:01.015088 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:01.015074 2577 scope.go:117] "RemoveContainer" containerID="dbcbd4b4f02ae7d04f4745e153d503e7a3825fcdb6aa9ac14992b5ec6b599d91" Jul 16 21:45:01.015353 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:01.015337 2577 scope.go:117] "RemoveContainer" containerID="baa4c46659374a860096bbc0dcf15542c63011509e0f736190f9e517f86b847c" Jul 16 21:45:01.015569 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:01.015543 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v_odh-ai-gateway-infra(fda983e2-daa2-46c5-89c9-02a367dfcf67)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v" podUID="fda983e2-daa2-46c5-89c9-02a367dfcf67" Jul 16 21:45:01.016900 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:01.016882 2577 generic.go:356] "Generic (PLEG): container finished" podID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" exitCode=1 Jul 16 21:45:01.017042 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:01.016949 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" event={"ID":"454a4f80-c2dd-45b4-9b59-9152db8367d1","Type":"ContainerDied","Data":"fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249"} Jul 16 21:45:01.017128 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:01.017117 2577 scope.go:117] "RemoveContainer" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" Jul 16 21:45:01.017310 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:01.017292 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:45:01.037543 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:01.037522 2577 scope.go:117] "RemoveContainer" containerID="d5150007b0c84271b523700a9cd8beb66ca8450f2320180bc70c5a049215d710" Jul 16 21:45:01.039387 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:01.039349 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737305-m79dl" podStartSLOduration=1.039334869 podStartE2EDuration="1.039334869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:45:01.037423114 +0000 UTC m=+2117.212531380" watchObservedRunningTime="2026-07-16 21:45:01.039334869 +0000 UTC m=+2117.214443061" Jul 16 21:45:01.156337 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:01.156286 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg" podStartSLOduration=1.156271599 podStartE2EDuration="1.156271599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:45:01.154475503 +0000 UTC m=+2117.329583742" watchObservedRunningTime="2026-07-16 21:45:01.156271599 +0000 UTC m=+2117.331379791" Jul 16 21:45:02.023774 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:02.023738 2577 generic.go:356] "Generic (PLEG): container finished" podID="3b80877d-921d-42d8-a8fa-74d0826123c3" containerID="c3decf7f6b29da77fbf1702d5fdbc00a45a91e6f8736a4d8ab5125f6e992daf7" exitCode=0 Jul 16 21:45:02.024244 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:02.023826 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737305-m79dl" event={"ID":"3b80877d-921d-42d8-a8fa-74d0826123c3","Type":"ContainerDied","Data":"c3decf7f6b29da77fbf1702d5fdbc00a45a91e6f8736a4d8ab5125f6e992daf7"} Jul 16 21:45:02.025276 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:02.025256 2577 generic.go:356] "Generic (PLEG): container finished" podID="89274d00-061a-4e73-9779-2bc6ffd22aa3" containerID="c3d56d00838da16874c8994bd0f74db92bc95274692519b070dad2af362a4ec5" exitCode=7 Jul 16 21:45:02.025371 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:02.025346 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg" event={"ID":"89274d00-061a-4e73-9779-2bc6ffd22aa3","Type":"ContainerDied","Data":"c3d56d00838da16874c8994bd0f74db92bc95274692519b070dad2af362a4ec5"} Jul 16 21:45:02.025485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:02.025472 2577 scope.go:117] "RemoveContainer" containerID="c3d56d00838da16874c8994bd0f74db92bc95274692519b070dad2af362a4ec5" Jul 16 21:45:02.028617 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:02.028602 2577 scope.go:117] "RemoveContainer" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" Jul 16 21:45:02.028764 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:02.028741 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:45:02.111944 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:02.111912 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v"] Jul 16 21:45:02.249579 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:02.249548 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v" Jul 16 21:45:02.363536 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:02.363501 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda983e2-daa2-46c5-89c9-02a367dfcf67-maas-api-service-ca\") pod \"fda983e2-daa2-46c5-89c9-02a367dfcf67\" (UID: \"fda983e2-daa2-46c5-89c9-02a367dfcf67\") " Jul 16 21:45:02.363868 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:02.363845 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda983e2-daa2-46c5-89c9-02a367dfcf67-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fda983e2-daa2-46c5-89c9-02a367dfcf67" (UID: "fda983e2-daa2-46c5-89c9-02a367dfcf67"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:45:02.464793 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:02.464768 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda983e2-daa2-46c5-89c9-02a367dfcf67-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:45:03.035906 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:03.035805 2577 generic.go:356] "Generic (PLEG): container finished" podID="89274d00-061a-4e73-9779-2bc6ffd22aa3" containerID="6b440f3524bc87fc1f2fc35ad5a9f47f0cc90f883e01dbdef8697f863c7a2b3f" exitCode=7 Jul 16 21:45:03.035906 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:03.035890 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg" event={"ID":"89274d00-061a-4e73-9779-2bc6ffd22aa3","Type":"ContainerDied","Data":"6b440f3524bc87fc1f2fc35ad5a9f47f0cc90f883e01dbdef8697f863c7a2b3f"} Jul 16 21:45:03.036494 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:03.035967 2577 scope.go:117] "RemoveContainer" containerID="c3d56d00838da16874c8994bd0f74db92bc95274692519b070dad2af362a4ec5" Jul 16 21:45:03.036494 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:03.036100 2577 scope.go:117] "RemoveContainer" containerID="6b440f3524bc87fc1f2fc35ad5a9f47f0cc90f883e01dbdef8697f863c7a2b3f" Jul 16 21:45:03.036494 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:03.036373 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg_odh-ai-gateway-infra(89274d00-061a-4e73-9779-2bc6ffd22aa3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg" podUID="89274d00-061a-4e73-9779-2bc6ffd22aa3" Jul 16 21:45:03.037174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:03.037151 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v" event={"ID":"fda983e2-daa2-46c5-89c9-02a367dfcf67","Type":"ContainerDied","Data":"8b7cf571619536d5247e805f667a7114c22274fbb405859bdf56c91d3f2c392d"} Jul 16 21:45:03.037257 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:03.037161 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v" Jul 16 21:45:03.047711 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:03.047690 2577 scope.go:117] "RemoveContainer" containerID="baa4c46659374a860096bbc0dcf15542c63011509e0f736190f9e517f86b847c" Jul 16 21:45:03.102869 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:03.102838 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v"] Jul 16 21:45:03.110387 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:03.110361 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2mx2v"] Jul 16 21:45:03.166337 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:03.166309 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737305-m79dl" Jul 16 21:45:04.043944 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.043917 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737305-m79dl" Jul 16 21:45:04.043944 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.043932 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737305-m79dl" event={"ID":"3b80877d-921d-42d8-a8fa-74d0826123c3","Type":"ContainerDied","Data":"1df5e0651d160446aa86c103f89876166fcf44a4a9cbd7e50db94e0b06054763"} Jul 16 21:45:04.044435 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.043981 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1df5e0651d160446aa86c103f89876166fcf44a4a9cbd7e50db94e0b06054763" Jul 16 21:45:04.045643 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.045618 2577 scope.go:117] "RemoveContainer" containerID="6b440f3524bc87fc1f2fc35ad5a9f47f0cc90f883e01dbdef8697f863c7a2b3f" Jul 16 21:45:04.045871 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:04.045849 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg_odh-ai-gateway-infra(89274d00-061a-4e73-9779-2bc6ffd22aa3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg" podUID="89274d00-061a-4e73-9779-2bc6ffd22aa3" Jul 16 21:45:04.450507 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.450476 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fda983e2-daa2-46c5-89c9-02a367dfcf67" path="/var/lib/kubelet/pods/fda983e2-daa2-46c5-89c9-02a367dfcf67/volumes" Jul 16 21:45:04.688086 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.688055 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q"] Jul 16 21:45:04.688431 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.688418 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b80877d-921d-42d8-a8fa-74d0826123c3" containerName="cleanup" Jul 16 21:45:04.688489 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.688433 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b80877d-921d-42d8-a8fa-74d0826123c3" containerName="cleanup" Jul 16 21:45:04.688489 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.688449 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda983e2-daa2-46c5-89c9-02a367dfcf67" containerName="revoke-keys" Jul 16 21:45:04.688489 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.688454 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda983e2-daa2-46c5-89c9-02a367dfcf67" containerName="revoke-keys" Jul 16 21:45:04.688489 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.688467 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda983e2-daa2-46c5-89c9-02a367dfcf67" containerName="revoke-keys" Jul 16 21:45:04.688489 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.688472 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda983e2-daa2-46c5-89c9-02a367dfcf67" containerName="revoke-keys" Jul 16 21:45:04.688489 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.688480 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda983e2-daa2-46c5-89c9-02a367dfcf67" containerName="revoke-keys" Jul 16 21:45:04.688489 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.688486 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda983e2-daa2-46c5-89c9-02a367dfcf67" containerName="revoke-keys" Jul 16 21:45:04.688702 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.688542 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fda983e2-daa2-46c5-89c9-02a367dfcf67" containerName="revoke-keys" Jul 16 21:45:04.688702 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.688552 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fda983e2-daa2-46c5-89c9-02a367dfcf67" containerName="revoke-keys" Jul 16 21:45:04.688702 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.688557 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b80877d-921d-42d8-a8fa-74d0826123c3" containerName="cleanup" Jul 16 21:45:04.691557 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.691541 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" Jul 16 21:45:04.734839 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.734754 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q"] Jul 16 21:45:04.783545 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.783504 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5c20df-bb78-480b-aadd-7efe1bf19d37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q\" (UID: \"3a5c20df-bb78-480b-aadd-7efe1bf19d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" Jul 16 21:45:04.884841 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.884804 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5c20df-bb78-480b-aadd-7efe1bf19d37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q\" (UID: \"3a5c20df-bb78-480b-aadd-7efe1bf19d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" Jul 16 21:45:04.885454 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:04.885436 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5c20df-bb78-480b-aadd-7efe1bf19d37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q\" (UID: \"3a5c20df-bb78-480b-aadd-7efe1bf19d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" Jul 16 21:45:05.001063 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:05.000944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" Jul 16 21:45:05.177952 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:05.177929 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q"] Jul 16 21:45:05.180300 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:45:05.180272 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5c20df_bb78_480b_aadd_7efe1bf19d37.slice/crio-88ddd9a4a10ceedd3a38f72d0f609306065133c23cc8edb1d433dd9d8f878058 WatchSource:0}: Error finding container 88ddd9a4a10ceedd3a38f72d0f609306065133c23cc8edb1d433dd9d8f878058: Status 404 returned error can't find the container with id 88ddd9a4a10ceedd3a38f72d0f609306065133c23cc8edb1d433dd9d8f878058 Jul 16 21:45:06.059698 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:06.059592 2577 generic.go:356] "Generic (PLEG): container finished" podID="3a5c20df-bb78-480b-aadd-7efe1bf19d37" containerID="bffe6257079b82b5354303e8aa0a614ef3507e767a03b3afe72c57d6d35a1bf3" exitCode=7 Jul 16 21:45:06.059861 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:06.059691 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" event={"ID":"3a5c20df-bb78-480b-aadd-7efe1bf19d37","Type":"ContainerDied","Data":"bffe6257079b82b5354303e8aa0a614ef3507e767a03b3afe72c57d6d35a1bf3"} Jul 16 21:45:06.059861 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:06.059745 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" event={"ID":"3a5c20df-bb78-480b-aadd-7efe1bf19d37","Type":"ContainerStarted","Data":"88ddd9a4a10ceedd3a38f72d0f609306065133c23cc8edb1d433dd9d8f878058"} Jul 16 21:45:06.059989 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:06.059873 2577 scope.go:117] "RemoveContainer" containerID="bffe6257079b82b5354303e8aa0a614ef3507e767a03b3afe72c57d6d35a1bf3" Jul 16 21:45:07.067383 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:07.067291 2577 generic.go:356] "Generic (PLEG): container finished" podID="3a5c20df-bb78-480b-aadd-7efe1bf19d37" containerID="e7036f5a2c1a2fd83ba17beff9f42e5e064e72c4bc8aa7190131a0debbe93597" exitCode=7 Jul 16 21:45:07.067383 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:07.067339 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" event={"ID":"3a5c20df-bb78-480b-aadd-7efe1bf19d37","Type":"ContainerDied","Data":"e7036f5a2c1a2fd83ba17beff9f42e5e064e72c4bc8aa7190131a0debbe93597"} Jul 16 21:45:07.067879 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:07.067394 2577 scope.go:117] "RemoveContainer" containerID="bffe6257079b82b5354303e8aa0a614ef3507e767a03b3afe72c57d6d35a1bf3" Jul 16 21:45:07.067879 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:07.067589 2577 scope.go:117] "RemoveContainer" containerID="e7036f5a2c1a2fd83ba17beff9f42e5e064e72c4bc8aa7190131a0debbe93597" Jul 16 21:45:07.067879 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:07.067828 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q_odh-ai-gateway-infra(3a5c20df-bb78-480b-aadd-7efe1bf19d37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" podUID="3a5c20df-bb78-480b-aadd-7efe1bf19d37" Jul 16 21:45:08.074081 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:08.074052 2577 scope.go:117] "RemoveContainer" containerID="e7036f5a2c1a2fd83ba17beff9f42e5e064e72c4bc8aa7190131a0debbe93597" Jul 16 21:45:08.074485 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:08.074276 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q_odh-ai-gateway-infra(3a5c20df-bb78-480b-aadd-7efe1bf19d37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" podUID="3a5c20df-bb78-480b-aadd-7efe1bf19d37" Jul 16 21:45:10.403914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:10.403875 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:45:10.404371 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:10.404355 2577 scope.go:117] "RemoveContainer" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" Jul 16 21:45:10.404541 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:10.404521 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:45:15.367074 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:15.367036 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx"] Jul 16 21:45:15.367558 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:15.367480 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fda983e2-daa2-46c5-89c9-02a367dfcf67" containerName="revoke-keys" Jul 16 21:45:15.371350 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:15.371332 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx" Jul 16 21:45:15.394212 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:15.394154 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx"] Jul 16 21:45:15.477972 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:15.477936 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e64ca7-04a6-435a-ba18-5446390a5ab2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx\" (UID: \"41e64ca7-04a6-435a-ba18-5446390a5ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx" Jul 16 21:45:15.578483 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:15.578448 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e64ca7-04a6-435a-ba18-5446390a5ab2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx\" (UID: \"41e64ca7-04a6-435a-ba18-5446390a5ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx" Jul 16 21:45:15.579114 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:15.579095 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e64ca7-04a6-435a-ba18-5446390a5ab2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx\" (UID: \"41e64ca7-04a6-435a-ba18-5446390a5ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx" Jul 16 21:45:15.681508 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:15.681458 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx" Jul 16 21:45:15.868749 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:15.868719 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx"] Jul 16 21:45:15.868978 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:45:15.868958 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e64ca7_04a6_435a_ba18_5446390a5ab2.slice/crio-1e70c95d993340fa4f7b0d8025edda21bb5620a5ceaa812fe6a2d05abdbfb05e WatchSource:0}: Error finding container 1e70c95d993340fa4f7b0d8025edda21bb5620a5ceaa812fe6a2d05abdbfb05e: Status 404 returned error can't find the container with id 1e70c95d993340fa4f7b0d8025edda21bb5620a5ceaa812fe6a2d05abdbfb05e Jul 16 21:45:16.114771 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:16.114733 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx" event={"ID":"41e64ca7-04a6-435a-ba18-5446390a5ab2","Type":"ContainerStarted","Data":"edcea9dac99f9fb9e0c3ed4e488686e9c9dbe45b281bbcf34fe193c16dff6207"} Jul 16 21:45:16.114877 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:16.114802 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx" event={"ID":"41e64ca7-04a6-435a-ba18-5446390a5ab2","Type":"ContainerStarted","Data":"1e70c95d993340fa4f7b0d8025edda21bb5620a5ceaa812fe6a2d05abdbfb05e"} Jul 16 21:45:17.121409 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:17.121367 2577 generic.go:356] "Generic (PLEG): container finished" podID="41e64ca7-04a6-435a-ba18-5446390a5ab2" containerID="edcea9dac99f9fb9e0c3ed4e488686e9c9dbe45b281bbcf34fe193c16dff6207" exitCode=6 Jul 16 21:45:17.121858 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:17.121423 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx" event={"ID":"41e64ca7-04a6-435a-ba18-5446390a5ab2","Type":"ContainerDied","Data":"edcea9dac99f9fb9e0c3ed4e488686e9c9dbe45b281bbcf34fe193c16dff6207"} Jul 16 21:45:17.121858 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:17.121680 2577 scope.go:117] "RemoveContainer" containerID="edcea9dac99f9fb9e0c3ed4e488686e9c9dbe45b281bbcf34fe193c16dff6207" Jul 16 21:45:18.131622 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:18.131586 2577 generic.go:356] "Generic (PLEG): container finished" podID="41e64ca7-04a6-435a-ba18-5446390a5ab2" containerID="bcb83c92d78839ddbc9e7f2524b6126251bf7a6561d1e38a504d814e2afdba13" exitCode=6 Jul 16 21:45:18.132022 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:18.131680 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx" event={"ID":"41e64ca7-04a6-435a-ba18-5446390a5ab2","Type":"ContainerDied","Data":"bcb83c92d78839ddbc9e7f2524b6126251bf7a6561d1e38a504d814e2afdba13"} Jul 16 21:45:18.132022 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:18.131788 2577 scope.go:117] "RemoveContainer" containerID="edcea9dac99f9fb9e0c3ed4e488686e9c9dbe45b281bbcf34fe193c16dff6207" Jul 16 21:45:18.132022 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:18.131839 2577 scope.go:117] "RemoveContainer" containerID="bcb83c92d78839ddbc9e7f2524b6126251bf7a6561d1e38a504d814e2afdba13" Jul 16 21:45:18.132164 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:18.132101 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx_odh-ai-gateway-infra(41e64ca7-04a6-435a-ba18-5446390a5ab2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx" podUID="41e64ca7-04a6-435a-ba18-5446390a5ab2" Jul 16 21:45:19.139214 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:19.139168 2577 scope.go:117] "RemoveContainer" containerID="bcb83c92d78839ddbc9e7f2524b6126251bf7a6561d1e38a504d814e2afdba13" Jul 16 21:45:19.139602 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:19.139378 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx_odh-ai-gateway-infra(41e64ca7-04a6-435a-ba18-5446390a5ab2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx" podUID="41e64ca7-04a6-435a-ba18-5446390a5ab2" Jul 16 21:45:19.444973 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:19.444894 2577 scope.go:117] "RemoveContainer" containerID="6b440f3524bc87fc1f2fc35ad5a9f47f0cc90f883e01dbdef8697f863c7a2b3f" Jul 16 21:45:20.146161 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:20.146130 2577 generic.go:356] "Generic (PLEG): container finished" podID="89274d00-061a-4e73-9779-2bc6ffd22aa3" containerID="e9d957b449f9918ef53abc07ed95c5a94857091aeeadad1e144d86e972ae7a69" exitCode=7 Jul 16 21:45:20.146597 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:20.146179 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg" event={"ID":"89274d00-061a-4e73-9779-2bc6ffd22aa3","Type":"ContainerDied","Data":"e9d957b449f9918ef53abc07ed95c5a94857091aeeadad1e144d86e972ae7a69"} Jul 16 21:45:20.146597 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:20.146256 2577 scope.go:117] "RemoveContainer" containerID="6b440f3524bc87fc1f2fc35ad5a9f47f0cc90f883e01dbdef8697f863c7a2b3f" Jul 16 21:45:20.146597 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:20.146524 2577 scope.go:117] "RemoveContainer" containerID="e9d957b449f9918ef53abc07ed95c5a94857091aeeadad1e144d86e972ae7a69" Jul 16 21:45:20.146783 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:20.146761 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg_odh-ai-gateway-infra(89274d00-061a-4e73-9779-2bc6ffd22aa3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg" podUID="89274d00-061a-4e73-9779-2bc6ffd22aa3" Jul 16 21:45:20.445864 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:20.445829 2577 scope.go:117] "RemoveContainer" containerID="e7036f5a2c1a2fd83ba17beff9f42e5e064e72c4bc8aa7190131a0debbe93597" Jul 16 21:45:21.154368 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:21.154334 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" event={"ID":"3a5c20df-bb78-480b-aadd-7efe1bf19d37","Type":"ContainerStarted","Data":"3bca8f6715068ffac97fecffc8f15d2a173cd6c5c2944751925d21b0511badba"} Jul 16 21:45:21.205014 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:21.204969 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" podStartSLOduration=17.204952609 podStartE2EDuration="17.204952609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:45:21.203758497 +0000 UTC m=+2137.378866702" watchObservedRunningTime="2026-07-16 21:45:21.204952609 +0000 UTC m=+2137.380060795" Jul 16 21:45:21.252500 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:21.252469 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg"] Jul 16 21:45:21.384929 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:21.384899 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg" Jul 16 21:45:21.445244 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:21.445125 2577 scope.go:117] "RemoveContainer" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" Jul 16 21:45:21.445419 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:21.445371 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:45:22.160925 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:22.160897 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg" Jul 16 21:45:22.161418 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:22.160942 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg" event={"ID":"89274d00-061a-4e73-9779-2bc6ffd22aa3","Type":"ContainerDied","Data":"171bbeff4ce7be72cb9ba48e57caf6d1f2b31c5bffe38c91b2664144efdc3370"} Jul 16 21:45:22.161418 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:22.161008 2577 scope.go:117] "RemoveContainer" containerID="e9d957b449f9918ef53abc07ed95c5a94857091aeeadad1e144d86e972ae7a69" Jul 16 21:45:22.163006 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:22.162963 2577 generic.go:356] "Generic (PLEG): container finished" podID="3a5c20df-bb78-480b-aadd-7efe1bf19d37" containerID="3bca8f6715068ffac97fecffc8f15d2a173cd6c5c2944751925d21b0511badba" exitCode=7 Jul 16 21:45:22.163140 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:22.163009 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" event={"ID":"3a5c20df-bb78-480b-aadd-7efe1bf19d37","Type":"ContainerDied","Data":"3bca8f6715068ffac97fecffc8f15d2a173cd6c5c2944751925d21b0511badba"} Jul 16 21:45:22.163362 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:22.163346 2577 scope.go:117] "RemoveContainer" containerID="3bca8f6715068ffac97fecffc8f15d2a173cd6c5c2944751925d21b0511badba" Jul 16 21:45:22.163542 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:22.163524 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q_odh-ai-gateway-infra(3a5c20df-bb78-480b-aadd-7efe1bf19d37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" podUID="3a5c20df-bb78-480b-aadd-7efe1bf19d37" Jul 16 21:45:22.170545 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:22.170528 2577 scope.go:117] "RemoveContainer" containerID="e7036f5a2c1a2fd83ba17beff9f42e5e064e72c4bc8aa7190131a0debbe93597" Jul 16 21:45:22.227965 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:22.227931 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q"] Jul 16 21:45:22.241735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:22.241700 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg"] Jul 16 21:45:22.250853 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:22.250827 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f60157c3-29737305-tpzvg"] Jul 16 21:45:22.450498 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:22.450414 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89274d00-061a-4e73-9779-2bc6ffd22aa3" path="/var/lib/kubelet/pods/89274d00-061a-4e73-9779-2bc6ffd22aa3/volumes" Jul 16 21:45:23.302922 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:23.302898 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" Jul 16 21:45:23.345614 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:23.345584 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5c20df-bb78-480b-aadd-7efe1bf19d37-maas-api-service-ca\") pod \"3a5c20df-bb78-480b-aadd-7efe1bf19d37\" (UID: \"3a5c20df-bb78-480b-aadd-7efe1bf19d37\") " Jul 16 21:45:23.345937 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:23.345916 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5c20df-bb78-480b-aadd-7efe1bf19d37-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a5c20df-bb78-480b-aadd-7efe1bf19d37" (UID: "3a5c20df-bb78-480b-aadd-7efe1bf19d37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:45:23.446440 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:23.446351 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5c20df-bb78-480b-aadd-7efe1bf19d37-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:45:24.181116 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:24.181086 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" Jul 16 21:45:24.181116 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:24.181109 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q" event={"ID":"3a5c20df-bb78-480b-aadd-7efe1bf19d37","Type":"ContainerDied","Data":"88ddd9a4a10ceedd3a38f72d0f609306065133c23cc8edb1d433dd9d8f878058"} Jul 16 21:45:24.181400 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:24.181165 2577 scope.go:117] "RemoveContainer" containerID="3bca8f6715068ffac97fecffc8f15d2a173cd6c5c2944751925d21b0511badba" Jul 16 21:45:24.254564 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:24.254529 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q"] Jul 16 21:45:24.256661 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:24.256630 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-zbr6q"] Jul 16 21:45:24.451380 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:24.451299 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a5c20df-bb78-480b-aadd-7efe1bf19d37" path="/var/lib/kubelet/pods/3a5c20df-bb78-480b-aadd-7efe1bf19d37/volumes" Jul 16 21:45:33.445618 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:33.445585 2577 scope.go:117] "RemoveContainer" containerID="bcb83c92d78839ddbc9e7f2524b6126251bf7a6561d1e38a504d814e2afdba13" Jul 16 21:45:34.237597 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:34.237562 2577 generic.go:356] "Generic (PLEG): container finished" podID="41e64ca7-04a6-435a-ba18-5446390a5ab2" containerID="47348f2afb8f13dfcae35f5e5f415062687694842ab0629f78c79b73570a0fd6" exitCode=6 Jul 16 21:45:34.237745 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:34.237616 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx" event={"ID":"41e64ca7-04a6-435a-ba18-5446390a5ab2","Type":"ContainerDied","Data":"47348f2afb8f13dfcae35f5e5f415062687694842ab0629f78c79b73570a0fd6"} Jul 16 21:45:34.237745 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:34.237682 2577 scope.go:117] "RemoveContainer" containerID="bcb83c92d78839ddbc9e7f2524b6126251bf7a6561d1e38a504d814e2afdba13" Jul 16 21:45:34.237983 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:34.237959 2577 scope.go:117] "RemoveContainer" containerID="47348f2afb8f13dfcae35f5e5f415062687694842ab0629f78c79b73570a0fd6" Jul 16 21:45:34.238258 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:34.238228 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx_odh-ai-gateway-infra(41e64ca7-04a6-435a-ba18-5446390a5ab2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx" podUID="41e64ca7-04a6-435a-ba18-5446390a5ab2" Jul 16 21:45:35.280563 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:35.280529 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx"] Jul 16 21:45:35.411837 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:35.411812 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx" Jul 16 21:45:35.453617 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:35.453584 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e64ca7-04a6-435a-ba18-5446390a5ab2-maas-api-service-ca\") pod \"41e64ca7-04a6-435a-ba18-5446390a5ab2\" (UID: \"41e64ca7-04a6-435a-ba18-5446390a5ab2\") " Jul 16 21:45:35.453960 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:35.453934 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e64ca7-04a6-435a-ba18-5446390a5ab2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41e64ca7-04a6-435a-ba18-5446390a5ab2" (UID: "41e64ca7-04a6-435a-ba18-5446390a5ab2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:45:35.554550 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:35.554470 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e64ca7-04a6-435a-ba18-5446390a5ab2-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:45:36.251803 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.251772 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx" Jul 16 21:45:36.251803 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.251787 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx" event={"ID":"41e64ca7-04a6-435a-ba18-5446390a5ab2","Type":"ContainerDied","Data":"1e70c95d993340fa4f7b0d8025edda21bb5620a5ceaa812fe6a2d05abdbfb05e"} Jul 16 21:45:36.252034 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.251843 2577 scope.go:117] "RemoveContainer" containerID="47348f2afb8f13dfcae35f5e5f415062687694842ab0629f78c79b73570a0fd6" Jul 16 21:45:36.282386 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.278349 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx"] Jul 16 21:45:36.283250 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.283228 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wrpvx"] Jul 16 21:45:36.445591 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.445556 2577 scope.go:117] "RemoveContainer" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" Jul 16 21:45:36.445808 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:36.445785 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:45:36.451088 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.451055 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41e64ca7-04a6-435a-ba18-5446390a5ab2" path="/var/lib/kubelet/pods/41e64ca7-04a6-435a-ba18-5446390a5ab2/volumes" Jul 16 21:45:36.516799 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.516718 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld"] Jul 16 21:45:36.517076 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517064 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e64ca7-04a6-435a-ba18-5446390a5ab2" containerName="revoke-keys" Jul 16 21:45:36.517076 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517078 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e64ca7-04a6-435a-ba18-5446390a5ab2" containerName="revoke-keys" Jul 16 21:45:36.517155 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517093 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89274d00-061a-4e73-9779-2bc6ffd22aa3" containerName="cleanup" Jul 16 21:45:36.517155 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517098 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89274d00-061a-4e73-9779-2bc6ffd22aa3" containerName="cleanup" Jul 16 21:45:36.517155 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517105 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e64ca7-04a6-435a-ba18-5446390a5ab2" containerName="revoke-keys" Jul 16 21:45:36.517155 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517111 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e64ca7-04a6-435a-ba18-5446390a5ab2" containerName="revoke-keys" Jul 16 21:45:36.517155 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517117 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5c20df-bb78-480b-aadd-7efe1bf19d37" containerName="revoke-keys" Jul 16 21:45:36.517155 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517122 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5c20df-bb78-480b-aadd-7efe1bf19d37" containerName="revoke-keys" Jul 16 21:45:36.517155 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517134 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89274d00-061a-4e73-9779-2bc6ffd22aa3" containerName="cleanup" Jul 16 21:45:36.517155 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517140 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89274d00-061a-4e73-9779-2bc6ffd22aa3" containerName="cleanup" Jul 16 21:45:36.517155 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517146 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5c20df-bb78-480b-aadd-7efe1bf19d37" containerName="revoke-keys" Jul 16 21:45:36.517155 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517151 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5c20df-bb78-480b-aadd-7efe1bf19d37" containerName="revoke-keys" Jul 16 21:45:36.517155 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517157 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89274d00-061a-4e73-9779-2bc6ffd22aa3" containerName="cleanup" Jul 16 21:45:36.517511 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517162 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89274d00-061a-4e73-9779-2bc6ffd22aa3" containerName="cleanup" Jul 16 21:45:36.517511 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517168 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5c20df-bb78-480b-aadd-7efe1bf19d37" containerName="revoke-keys" Jul 16 21:45:36.517511 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517173 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5c20df-bb78-480b-aadd-7efe1bf19d37" containerName="revoke-keys" Jul 16 21:45:36.517511 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517241 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e64ca7-04a6-435a-ba18-5446390a5ab2" containerName="revoke-keys" Jul 16 21:45:36.517511 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517248 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5c20df-bb78-480b-aadd-7efe1bf19d37" containerName="revoke-keys" Jul 16 21:45:36.517511 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517254 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89274d00-061a-4e73-9779-2bc6ffd22aa3" containerName="cleanup" Jul 16 21:45:36.517511 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517261 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5c20df-bb78-480b-aadd-7efe1bf19d37" containerName="revoke-keys" Jul 16 21:45:36.517511 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517266 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89274d00-061a-4e73-9779-2bc6ffd22aa3" containerName="cleanup" Jul 16 21:45:36.517511 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517272 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e64ca7-04a6-435a-ba18-5446390a5ab2" containerName="revoke-keys" Jul 16 21:45:36.517511 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.517276 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89274d00-061a-4e73-9779-2bc6ffd22aa3" containerName="cleanup" Jul 16 21:45:36.521550 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.521534 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" Jul 16 21:45:36.525513 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.525486 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:45:36.532631 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.532607 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld"] Jul 16 21:45:36.564398 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.564357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8ce713-ca09-46ff-a5ff-b15e8d231297-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld\" (UID: \"8f8ce713-ca09-46ff-a5ff-b15e8d231297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" Jul 16 21:45:36.665382 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.665346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8ce713-ca09-46ff-a5ff-b15e8d231297-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld\" (UID: \"8f8ce713-ca09-46ff-a5ff-b15e8d231297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" Jul 16 21:45:36.665979 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.665957 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8ce713-ca09-46ff-a5ff-b15e8d231297-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld\" (UID: \"8f8ce713-ca09-46ff-a5ff-b15e8d231297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" Jul 16 21:45:36.832941 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.832852 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" Jul 16 21:45:36.972163 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:36.972137 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld"] Jul 16 21:45:36.974021 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:45:36.973994 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8ce713_ca09_46ff_a5ff_b15e8d231297.slice/crio-d85f2ee56a298a9b7b859852de2c9a5fce703c678f5f7081c6ca38deb12f026d WatchSource:0}: Error finding container d85f2ee56a298a9b7b859852de2c9a5fce703c678f5f7081c6ca38deb12f026d: Status 404 returned error can't find the container with id d85f2ee56a298a9b7b859852de2c9a5fce703c678f5f7081c6ca38deb12f026d Jul 16 21:45:37.258392 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:37.258350 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" event={"ID":"8f8ce713-ca09-46ff-a5ff-b15e8d231297","Type":"ContainerStarted","Data":"23f77818c474e083672203c1cb6ffa56f4c6e5fe47cd1c445bd8628efa3d8514"} Jul 16 21:45:37.258581 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:37.258412 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" event={"ID":"8f8ce713-ca09-46ff-a5ff-b15e8d231297","Type":"ContainerStarted","Data":"d85f2ee56a298a9b7b859852de2c9a5fce703c678f5f7081c6ca38deb12f026d"} Jul 16 21:45:37.284736 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:37.284681 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" podStartSLOduration=1.284664953 podStartE2EDuration="1.284664953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:45:37.283887769 +0000 UTC m=+2153.458995961" watchObservedRunningTime="2026-07-16 21:45:37.284664953 +0000 UTC m=+2153.459773144" Jul 16 21:45:38.266419 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:38.266385 2577 generic.go:356] "Generic (PLEG): container finished" podID="8f8ce713-ca09-46ff-a5ff-b15e8d231297" containerID="23f77818c474e083672203c1cb6ffa56f4c6e5fe47cd1c445bd8628efa3d8514" exitCode=6 Jul 16 21:45:38.266608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:38.266468 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" event={"ID":"8f8ce713-ca09-46ff-a5ff-b15e8d231297","Type":"ContainerDied","Data":"23f77818c474e083672203c1cb6ffa56f4c6e5fe47cd1c445bd8628efa3d8514"} Jul 16 21:45:38.266669 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:38.266654 2577 scope.go:117] "RemoveContainer" containerID="23f77818c474e083672203c1cb6ffa56f4c6e5fe47cd1c445bd8628efa3d8514" Jul 16 21:45:39.273701 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:39.273614 2577 generic.go:356] "Generic (PLEG): container finished" podID="8f8ce713-ca09-46ff-a5ff-b15e8d231297" containerID="d3ed4a142bf09de47af33c139d86a1b3006b67b02e45534d67565f3e44e5f886" exitCode=6 Jul 16 21:45:39.273701 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:39.273670 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" event={"ID":"8f8ce713-ca09-46ff-a5ff-b15e8d231297","Type":"ContainerDied","Data":"d3ed4a142bf09de47af33c139d86a1b3006b67b02e45534d67565f3e44e5f886"} Jul 16 21:45:39.274179 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:39.273721 2577 scope.go:117] "RemoveContainer" containerID="23f77818c474e083672203c1cb6ffa56f4c6e5fe47cd1c445bd8628efa3d8514" Jul 16 21:45:39.274179 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:39.273902 2577 scope.go:117] "RemoveContainer" containerID="d3ed4a142bf09de47af33c139d86a1b3006b67b02e45534d67565f3e44e5f886" Jul 16 21:45:39.274179 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:39.274123 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld_odh-ai-gateway-infra(8f8ce713-ca09-46ff-a5ff-b15e8d231297)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" podUID="8f8ce713-ca09-46ff-a5ff-b15e8d231297" Jul 16 21:45:40.280943 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:40.280916 2577 scope.go:117] "RemoveContainer" containerID="d3ed4a142bf09de47af33c139d86a1b3006b67b02e45534d67565f3e44e5f886" Jul 16 21:45:40.281335 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:40.281101 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld_odh-ai-gateway-infra(8f8ce713-ca09-46ff-a5ff-b15e8d231297)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" podUID="8f8ce713-ca09-46ff-a5ff-b15e8d231297" Jul 16 21:45:50.445052 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:50.445016 2577 scope.go:117] "RemoveContainer" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" Jul 16 21:45:50.445585 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:50.445283 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:45:52.445072 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:52.445040 2577 scope.go:117] "RemoveContainer" containerID="d3ed4a142bf09de47af33c139d86a1b3006b67b02e45534d67565f3e44e5f886" Jul 16 21:45:53.350722 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:53.350626 2577 generic.go:356] "Generic (PLEG): container finished" podID="8f8ce713-ca09-46ff-a5ff-b15e8d231297" containerID="631af87601997868198701d3560ebde6ffc41c72f40871b8c4d11de31d0a3a2b" exitCode=6 Jul 16 21:45:53.350722 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:53.350680 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" event={"ID":"8f8ce713-ca09-46ff-a5ff-b15e8d231297","Type":"ContainerDied","Data":"631af87601997868198701d3560ebde6ffc41c72f40871b8c4d11de31d0a3a2b"} Jul 16 21:45:53.350942 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:53.350758 2577 scope.go:117] "RemoveContainer" containerID="d3ed4a142bf09de47af33c139d86a1b3006b67b02e45534d67565f3e44e5f886" Jul 16 21:45:53.351111 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:53.351093 2577 scope.go:117] "RemoveContainer" containerID="631af87601997868198701d3560ebde6ffc41c72f40871b8c4d11de31d0a3a2b" Jul 16 21:45:53.351358 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:53.351335 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld_odh-ai-gateway-infra(8f8ce713-ca09-46ff-a5ff-b15e8d231297)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" podUID="8f8ce713-ca09-46ff-a5ff-b15e8d231297" Jul 16 21:45:54.382112 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.382079 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld"] Jul 16 21:45:54.536726 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.536702 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" Jul 16 21:45:54.624474 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.624441 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8ce713-ca09-46ff-a5ff-b15e8d231297-maas-api-service-ca\") pod \"8f8ce713-ca09-46ff-a5ff-b15e8d231297\" (UID: \"8f8ce713-ca09-46ff-a5ff-b15e8d231297\") " Jul 16 21:45:54.624865 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.624836 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8ce713-ca09-46ff-a5ff-b15e8d231297-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f8ce713-ca09-46ff-a5ff-b15e8d231297" (UID: "8f8ce713-ca09-46ff-a5ff-b15e8d231297"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:45:54.725705 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.725627 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8ce713-ca09-46ff-a5ff-b15e8d231297-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:45:54.747974 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.747945 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8"] Jul 16 21:45:54.748323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.748310 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8ce713-ca09-46ff-a5ff-b15e8d231297" containerName="revoke-keys" Jul 16 21:45:54.748323 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.748324 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8ce713-ca09-46ff-a5ff-b15e8d231297" containerName="revoke-keys" Jul 16 21:45:54.748453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.748332 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e64ca7-04a6-435a-ba18-5446390a5ab2" containerName="revoke-keys" Jul 16 21:45:54.748453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.748337 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e64ca7-04a6-435a-ba18-5446390a5ab2" containerName="revoke-keys" Jul 16 21:45:54.748453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.748345 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8ce713-ca09-46ff-a5ff-b15e8d231297" containerName="revoke-keys" Jul 16 21:45:54.748453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.748350 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8ce713-ca09-46ff-a5ff-b15e8d231297" containerName="revoke-keys" Jul 16 21:45:54.748453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.748358 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8ce713-ca09-46ff-a5ff-b15e8d231297" containerName="revoke-keys" Jul 16 21:45:54.748453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.748364 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8ce713-ca09-46ff-a5ff-b15e8d231297" containerName="revoke-keys" Jul 16 21:45:54.748453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.748449 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5c20df-bb78-480b-aadd-7efe1bf19d37" containerName="revoke-keys" Jul 16 21:45:54.748715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.748457 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f8ce713-ca09-46ff-a5ff-b15e8d231297" containerName="revoke-keys" Jul 16 21:45:54.748715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.748464 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e64ca7-04a6-435a-ba18-5446390a5ab2" containerName="revoke-keys" Jul 16 21:45:54.748715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.748470 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f8ce713-ca09-46ff-a5ff-b15e8d231297" containerName="revoke-keys" Jul 16 21:45:54.748715 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.748479 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f8ce713-ca09-46ff-a5ff-b15e8d231297" containerName="revoke-keys" Jul 16 21:45:54.751371 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.751356 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8" Jul 16 21:45:54.760035 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.760014 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8"] Jul 16 21:45:54.827075 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.827036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48fbd53c-9fa3-4104-ba4e-c0ac95640cdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8\" (UID: \"48fbd53c-9fa3-4104-ba4e-c0ac95640cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8" Jul 16 21:45:54.928059 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.928024 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48fbd53c-9fa3-4104-ba4e-c0ac95640cdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8\" (UID: \"48fbd53c-9fa3-4104-ba4e-c0ac95640cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8" Jul 16 21:45:54.928663 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:54.928642 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48fbd53c-9fa3-4104-ba4e-c0ac95640cdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8\" (UID: \"48fbd53c-9fa3-4104-ba4e-c0ac95640cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8" Jul 16 21:45:55.062779 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:55.062691 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8" Jul 16 21:45:55.211609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:55.211577 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8"] Jul 16 21:45:55.215620 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:45:55.215595 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fbd53c_9fa3_4104_ba4e_c0ac95640cdf.slice/crio-2089b060da03c9b00991367564eb2d7da98f336676dff26a8214d15ff60af661 WatchSource:0}: Error finding container 2089b060da03c9b00991367564eb2d7da98f336676dff26a8214d15ff60af661: Status 404 returned error can't find the container with id 2089b060da03c9b00991367564eb2d7da98f336676dff26a8214d15ff60af661 Jul 16 21:45:55.364907 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:55.364827 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" Jul 16 21:45:55.364907 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:55.364874 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld" event={"ID":"8f8ce713-ca09-46ff-a5ff-b15e8d231297","Type":"ContainerDied","Data":"d85f2ee56a298a9b7b859852de2c9a5fce703c678f5f7081c6ca38deb12f026d"} Jul 16 21:45:55.365130 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:55.364948 2577 scope.go:117] "RemoveContainer" containerID="631af87601997868198701d3560ebde6ffc41c72f40871b8c4d11de31d0a3a2b" Jul 16 21:45:55.366276 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:55.366246 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8" event={"ID":"48fbd53c-9fa3-4104-ba4e-c0ac95640cdf","Type":"ContainerStarted","Data":"2089b060da03c9b00991367564eb2d7da98f336676dff26a8214d15ff60af661"} Jul 16 21:45:55.391727 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:55.391702 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld"] Jul 16 21:45:55.397465 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:55.397442 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-ltcld"] Jul 16 21:45:56.372455 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:56.372422 2577 generic.go:356] "Generic (PLEG): container finished" podID="48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" containerID="c94e7e54fe38705db43b0c0dcd8835032574d9cead7f10a5565ca8010b84ac2a" exitCode=7 Jul 16 21:45:56.372656 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:56.372504 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8" event={"ID":"48fbd53c-9fa3-4104-ba4e-c0ac95640cdf","Type":"ContainerDied","Data":"c94e7e54fe38705db43b0c0dcd8835032574d9cead7f10a5565ca8010b84ac2a"} Jul 16 21:45:56.372724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:56.372683 2577 scope.go:117] "RemoveContainer" containerID="c94e7e54fe38705db43b0c0dcd8835032574d9cead7f10a5565ca8010b84ac2a" Jul 16 21:45:56.450479 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:56.450455 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f8ce713-ca09-46ff-a5ff-b15e8d231297" path="/var/lib/kubelet/pods/8f8ce713-ca09-46ff-a5ff-b15e8d231297/volumes" Jul 16 21:45:57.386966 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:57.386933 2577 generic.go:356] "Generic (PLEG): container finished" podID="48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" containerID="820d55b97bb1afc2b0192bbdb807867a9bc8e52e691b3dc1d25374e5909e95d4" exitCode=7 Jul 16 21:45:57.387121 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:57.386987 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8" event={"ID":"48fbd53c-9fa3-4104-ba4e-c0ac95640cdf","Type":"ContainerDied","Data":"820d55b97bb1afc2b0192bbdb807867a9bc8e52e691b3dc1d25374e5909e95d4"} Jul 16 21:45:57.387121 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:57.387081 2577 scope.go:117] "RemoveContainer" containerID="c94e7e54fe38705db43b0c0dcd8835032574d9cead7f10a5565ca8010b84ac2a" Jul 16 21:45:57.387278 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:57.387267 2577 scope.go:117] "RemoveContainer" containerID="820d55b97bb1afc2b0192bbdb807867a9bc8e52e691b3dc1d25374e5909e95d4" Jul 16 21:45:57.387495 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:57.387475 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8_odh-ai-gateway-infra(48fbd53c-9fa3-4104-ba4e-c0ac95640cdf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8" podUID="48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" Jul 16 21:45:58.393895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:45:58.393869 2577 scope.go:117] "RemoveContainer" containerID="820d55b97bb1afc2b0192bbdb807867a9bc8e52e691b3dc1d25374e5909e95d4" Jul 16 21:45:58.394295 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:45:58.394064 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8_odh-ai-gateway-infra(48fbd53c-9fa3-4104-ba4e-c0ac95640cdf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8" podUID="48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" Jul 16 21:46:05.445658 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:05.445619 2577 scope.go:117] "RemoveContainer" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" Jul 16 21:46:05.446122 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:46:05.445850 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:46:10.445693 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:10.445663 2577 scope.go:117] "RemoveContainer" containerID="820d55b97bb1afc2b0192bbdb807867a9bc8e52e691b3dc1d25374e5909e95d4" Jul 16 21:46:11.460624 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:11.460537 2577 generic.go:356] "Generic (PLEG): container finished" podID="48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" containerID="4a4200978a66cdf7647944e3c46d7afb0b47d046f2459948d751ed29644830a5" exitCode=7 Jul 16 21:46:11.460624 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:11.460597 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8" event={"ID":"48fbd53c-9fa3-4104-ba4e-c0ac95640cdf","Type":"ContainerDied","Data":"4a4200978a66cdf7647944e3c46d7afb0b47d046f2459948d751ed29644830a5"} Jul 16 21:46:11.461067 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:11.460673 2577 scope.go:117] "RemoveContainer" containerID="820d55b97bb1afc2b0192bbdb807867a9bc8e52e691b3dc1d25374e5909e95d4" Jul 16 21:46:11.461067 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:11.460991 2577 scope.go:117] "RemoveContainer" containerID="4a4200978a66cdf7647944e3c46d7afb0b47d046f2459948d751ed29644830a5" Jul 16 21:46:11.461270 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:46:11.461250 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8_odh-ai-gateway-infra(48fbd53c-9fa3-4104-ba4e-c0ac95640cdf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8" podUID="48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" Jul 16 21:46:12.506726 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:12.506690 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8"] Jul 16 21:46:12.642895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:12.642868 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8" Jul 16 21:46:12.788408 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:12.788309 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48fbd53c-9fa3-4104-ba4e-c0ac95640cdf-maas-api-service-ca\") pod \"48fbd53c-9fa3-4104-ba4e-c0ac95640cdf\" (UID: \"48fbd53c-9fa3-4104-ba4e-c0ac95640cdf\") " Jul 16 21:46:12.788676 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:12.788652 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48fbd53c-9fa3-4104-ba4e-c0ac95640cdf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" (UID: "48fbd53c-9fa3-4104-ba4e-c0ac95640cdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:46:12.889888 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:12.889850 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48fbd53c-9fa3-4104-ba4e-c0ac95640cdf-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:46:13.472930 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:13.472895 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8" event={"ID":"48fbd53c-9fa3-4104-ba4e-c0ac95640cdf","Type":"ContainerDied","Data":"2089b060da03c9b00991367564eb2d7da98f336676dff26a8214d15ff60af661"} Jul 16 21:46:13.473129 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:13.472950 2577 scope.go:117] "RemoveContainer" containerID="4a4200978a66cdf7647944e3c46d7afb0b47d046f2459948d751ed29644830a5" Jul 16 21:46:13.473129 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:13.472908 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8" Jul 16 21:46:13.498485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:13.498455 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8"] Jul 16 21:46:13.502081 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:13.502058 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rzpq8"] Jul 16 21:46:14.450121 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:14.450085 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" path="/var/lib/kubelet/pods/48fbd53c-9fa3-4104-ba4e-c0ac95640cdf/volumes" Jul 16 21:46:15.396766 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.396734 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz"] Jul 16 21:46:15.397083 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.397072 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" containerName="revoke-keys" Jul 16 21:46:15.397125 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.397085 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" containerName="revoke-keys" Jul 16 21:46:15.397125 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.397094 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" containerName="revoke-keys" Jul 16 21:46:15.397125 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.397099 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" containerName="revoke-keys" Jul 16 21:46:15.397254 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.397138 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" containerName="revoke-keys" Jul 16 21:46:15.397254 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.397144 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" containerName="revoke-keys" Jul 16 21:46:15.397254 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.397218 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" containerName="revoke-keys" Jul 16 21:46:15.397254 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.397226 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" containerName="revoke-keys" Jul 16 21:46:15.397254 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.397233 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="48fbd53c-9fa3-4104-ba4e-c0ac95640cdf" containerName="revoke-keys" Jul 16 21:46:15.401382 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.401363 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz" Jul 16 21:46:15.404273 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.404249 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:46:15.410042 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.410015 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz"] Jul 16 21:46:15.414201 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.414149 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f067acf8-5e20-4e5e-aa97-1011a3c7717e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz\" (UID: \"f067acf8-5e20-4e5e-aa97-1011a3c7717e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz" Jul 16 21:46:15.514834 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.514795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f067acf8-5e20-4e5e-aa97-1011a3c7717e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz\" (UID: \"f067acf8-5e20-4e5e-aa97-1011a3c7717e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz" Jul 16 21:46:15.515478 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.515458 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f067acf8-5e20-4e5e-aa97-1011a3c7717e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz\" (UID: \"f067acf8-5e20-4e5e-aa97-1011a3c7717e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz" Jul 16 21:46:15.713384 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.713306 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz" Jul 16 21:46:15.852550 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:15.852519 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz"] Jul 16 21:46:15.854221 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:46:15.854176 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf067acf8_5e20_4e5e_aa97_1011a3c7717e.slice/crio-51179d98b554514df3fc7e35362e966a313bacc6792fe48f3163bc483f6e984f WatchSource:0}: Error finding container 51179d98b554514df3fc7e35362e966a313bacc6792fe48f3163bc483f6e984f: Status 404 returned error can't find the container with id 51179d98b554514df3fc7e35362e966a313bacc6792fe48f3163bc483f6e984f Jul 16 21:46:16.497975 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:16.497947 2577 generic.go:356] "Generic (PLEG): container finished" podID="f067acf8-5e20-4e5e-aa97-1011a3c7717e" containerID="6ad622cdd8677d9100c7975ad8be5358d64e12ff13193c32b041f861e3743953" exitCode=6 Jul 16 21:46:16.498110 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:16.498029 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz" event={"ID":"f067acf8-5e20-4e5e-aa97-1011a3c7717e","Type":"ContainerDied","Data":"6ad622cdd8677d9100c7975ad8be5358d64e12ff13193c32b041f861e3743953"} Jul 16 21:46:16.498110 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:16.498093 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz" event={"ID":"f067acf8-5e20-4e5e-aa97-1011a3c7717e","Type":"ContainerStarted","Data":"51179d98b554514df3fc7e35362e966a313bacc6792fe48f3163bc483f6e984f"} Jul 16 21:46:16.498211 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:16.498200 2577 scope.go:117] "RemoveContainer" containerID="6ad622cdd8677d9100c7975ad8be5358d64e12ff13193c32b041f861e3743953" Jul 16 21:46:17.445568 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:17.445538 2577 scope.go:117] "RemoveContainer" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" Jul 16 21:46:17.445919 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:46:17.445716 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:46:17.506059 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:17.506029 2577 generic.go:356] "Generic (PLEG): container finished" podID="f067acf8-5e20-4e5e-aa97-1011a3c7717e" containerID="f6a4ebf5ce5171c54f3f0be004aae49dfbf6b2ff0e6b2ce96396c96024671c5f" exitCode=6 Jul 16 21:46:17.506282 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:17.506107 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz" event={"ID":"f067acf8-5e20-4e5e-aa97-1011a3c7717e","Type":"ContainerDied","Data":"f6a4ebf5ce5171c54f3f0be004aae49dfbf6b2ff0e6b2ce96396c96024671c5f"} Jul 16 21:46:17.506282 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:17.506172 2577 scope.go:117] "RemoveContainer" containerID="6ad622cdd8677d9100c7975ad8be5358d64e12ff13193c32b041f861e3743953" Jul 16 21:46:17.506410 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:17.506387 2577 scope.go:117] "RemoveContainer" containerID="f6a4ebf5ce5171c54f3f0be004aae49dfbf6b2ff0e6b2ce96396c96024671c5f" Jul 16 21:46:17.506622 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:46:17.506600 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz_odh-ai-gateway-infra(f067acf8-5e20-4e5e-aa97-1011a3c7717e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz" podUID="f067acf8-5e20-4e5e-aa97-1011a3c7717e" Jul 16 21:46:18.513102 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:18.513075 2577 scope.go:117] "RemoveContainer" containerID="f6a4ebf5ce5171c54f3f0be004aae49dfbf6b2ff0e6b2ce96396c96024671c5f" Jul 16 21:46:18.513513 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:46:18.513367 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz_odh-ai-gateway-infra(f067acf8-5e20-4e5e-aa97-1011a3c7717e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz" podUID="f067acf8-5e20-4e5e-aa97-1011a3c7717e" Jul 16 21:46:26.543430 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:26.543391 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj"] Jul 16 21:46:26.546820 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:26.546801 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" Jul 16 21:46:26.555272 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:26.555201 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj"] Jul 16 21:46:26.616907 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:26.616861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23200d03-2bbb-49a2-a889-21ef6ef741de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj\" (UID: \"23200d03-2bbb-49a2-a889-21ef6ef741de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" Jul 16 21:46:26.718145 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:26.718103 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23200d03-2bbb-49a2-a889-21ef6ef741de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj\" (UID: \"23200d03-2bbb-49a2-a889-21ef6ef741de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" Jul 16 21:46:26.718795 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:26.718774 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23200d03-2bbb-49a2-a889-21ef6ef741de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj\" (UID: \"23200d03-2bbb-49a2-a889-21ef6ef741de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" Jul 16 21:46:26.859950 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:26.859847 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" Jul 16 21:46:26.995427 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:26.995397 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj"] Jul 16 21:46:26.997109 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:46:26.997080 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23200d03_2bbb_49a2_a889_21ef6ef741de.slice/crio-6615b4976c557b3037f47ed92335d5ec07056f7b0c6af0f376b051a13e805ba2 WatchSource:0}: Error finding container 6615b4976c557b3037f47ed92335d5ec07056f7b0c6af0f376b051a13e805ba2: Status 404 returned error can't find the container with id 6615b4976c557b3037f47ed92335d5ec07056f7b0c6af0f376b051a13e805ba2 Jul 16 21:46:27.560024 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:27.559932 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" event={"ID":"23200d03-2bbb-49a2-a889-21ef6ef741de","Type":"ContainerStarted","Data":"b2f09f4b9eab84d7c6ca0dc056a25e5aa5906240c2f73fa00cd1112358da4578"} Jul 16 21:46:27.560024 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:27.559986 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" event={"ID":"23200d03-2bbb-49a2-a889-21ef6ef741de","Type":"ContainerStarted","Data":"6615b4976c557b3037f47ed92335d5ec07056f7b0c6af0f376b051a13e805ba2"} Jul 16 21:46:27.577468 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:27.577398 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" podStartSLOduration=1.577376568 podStartE2EDuration="1.577376568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:46:27.575155825 +0000 UTC m=+2203.750264019" watchObservedRunningTime="2026-07-16 21:46:27.577376568 +0000 UTC m=+2203.752484761" Jul 16 21:46:28.567640 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:28.567600 2577 generic.go:356] "Generic (PLEG): container finished" podID="23200d03-2bbb-49a2-a889-21ef6ef741de" containerID="b2f09f4b9eab84d7c6ca0dc056a25e5aa5906240c2f73fa00cd1112358da4578" exitCode=6 Jul 16 21:46:28.568016 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:28.567690 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" event={"ID":"23200d03-2bbb-49a2-a889-21ef6ef741de","Type":"ContainerDied","Data":"b2f09f4b9eab84d7c6ca0dc056a25e5aa5906240c2f73fa00cd1112358da4578"} Jul 16 21:46:28.568016 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:28.567836 2577 scope.go:117] "RemoveContainer" containerID="b2f09f4b9eab84d7c6ca0dc056a25e5aa5906240c2f73fa00cd1112358da4578" Jul 16 21:46:29.445651 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:29.445569 2577 scope.go:117] "RemoveContainer" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" Jul 16 21:46:29.445858 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:46:29.445753 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:46:29.574276 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:29.574247 2577 generic.go:356] "Generic (PLEG): container finished" podID="23200d03-2bbb-49a2-a889-21ef6ef741de" containerID="1490056ff57b50b51df985f18ed272c2eff9431a50c22ae6b11b90abc22dff27" exitCode=6 Jul 16 21:46:29.574689 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:29.574338 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" event={"ID":"23200d03-2bbb-49a2-a889-21ef6ef741de","Type":"ContainerDied","Data":"1490056ff57b50b51df985f18ed272c2eff9431a50c22ae6b11b90abc22dff27"} Jul 16 21:46:29.574689 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:29.574393 2577 scope.go:117] "RemoveContainer" containerID="b2f09f4b9eab84d7c6ca0dc056a25e5aa5906240c2f73fa00cd1112358da4578" Jul 16 21:46:29.574689 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:29.574519 2577 scope.go:117] "RemoveContainer" containerID="1490056ff57b50b51df985f18ed272c2eff9431a50c22ae6b11b90abc22dff27" Jul 16 21:46:29.574829 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:46:29.574767 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj_odh-ai-gateway-infra(23200d03-2bbb-49a2-a889-21ef6ef741de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" podUID="23200d03-2bbb-49a2-a889-21ef6ef741de" Jul 16 21:46:30.445438 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:30.445397 2577 scope.go:117] "RemoveContainer" containerID="f6a4ebf5ce5171c54f3f0be004aae49dfbf6b2ff0e6b2ce96396c96024671c5f" Jul 16 21:46:30.581545 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:30.581517 2577 scope.go:117] "RemoveContainer" containerID="1490056ff57b50b51df985f18ed272c2eff9431a50c22ae6b11b90abc22dff27" Jul 16 21:46:30.581896 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:46:30.581715 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj_odh-ai-gateway-infra(23200d03-2bbb-49a2-a889-21ef6ef741de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" podUID="23200d03-2bbb-49a2-a889-21ef6ef741de" Jul 16 21:46:31.588718 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:31.588621 2577 generic.go:356] "Generic (PLEG): container finished" podID="f067acf8-5e20-4e5e-aa97-1011a3c7717e" containerID="5e8ea18b68eaf9ecc4c1b1bdea55aaa360694996bdbadd6c304f9afbea9f5186" exitCode=6 Jul 16 21:46:31.588718 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:31.588695 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz" event={"ID":"f067acf8-5e20-4e5e-aa97-1011a3c7717e","Type":"ContainerDied","Data":"5e8ea18b68eaf9ecc4c1b1bdea55aaa360694996bdbadd6c304f9afbea9f5186"} Jul 16 21:46:31.589219 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:31.588762 2577 scope.go:117] "RemoveContainer" containerID="f6a4ebf5ce5171c54f3f0be004aae49dfbf6b2ff0e6b2ce96396c96024671c5f" Jul 16 21:46:31.589219 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:31.589058 2577 scope.go:117] "RemoveContainer" containerID="5e8ea18b68eaf9ecc4c1b1bdea55aaa360694996bdbadd6c304f9afbea9f5186" Jul 16 21:46:31.589345 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:46:31.589258 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz_odh-ai-gateway-infra(f067acf8-5e20-4e5e-aa97-1011a3c7717e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz" podUID="f067acf8-5e20-4e5e-aa97-1011a3c7717e" Jul 16 21:46:32.618639 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:32.618606 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz"] Jul 16 21:46:32.756086 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:32.756061 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz" Jul 16 21:46:32.873743 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:32.873640 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f067acf8-5e20-4e5e-aa97-1011a3c7717e-maas-api-service-ca\") pod \"f067acf8-5e20-4e5e-aa97-1011a3c7717e\" (UID: \"f067acf8-5e20-4e5e-aa97-1011a3c7717e\") " Jul 16 21:46:32.874004 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:32.873983 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f067acf8-5e20-4e5e-aa97-1011a3c7717e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f067acf8-5e20-4e5e-aa97-1011a3c7717e" (UID: "f067acf8-5e20-4e5e-aa97-1011a3c7717e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:46:32.975171 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:32.975139 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f067acf8-5e20-4e5e-aa97-1011a3c7717e-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:46:33.610124 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:33.610093 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz" Jul 16 21:46:33.610124 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:33.610116 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz" event={"ID":"f067acf8-5e20-4e5e-aa97-1011a3c7717e","Type":"ContainerDied","Data":"51179d98b554514df3fc7e35362e966a313bacc6792fe48f3163bc483f6e984f"} Jul 16 21:46:33.610368 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:33.610172 2577 scope.go:117] "RemoveContainer" containerID="5e8ea18b68eaf9ecc4c1b1bdea55aaa360694996bdbadd6c304f9afbea9f5186" Jul 16 21:46:33.635230 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:33.635178 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz"] Jul 16 21:46:33.643095 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:33.643065 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-294hz"] Jul 16 21:46:34.450326 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:34.450297 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f067acf8-5e20-4e5e-aa97-1011a3c7717e" path="/var/lib/kubelet/pods/f067acf8-5e20-4e5e-aa97-1011a3c7717e/volumes" Jul 16 21:46:42.445493 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:42.445465 2577 scope.go:117] "RemoveContainer" containerID="1490056ff57b50b51df985f18ed272c2eff9431a50c22ae6b11b90abc22dff27" Jul 16 21:46:43.445453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:43.445423 2577 scope.go:117] "RemoveContainer" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" Jul 16 21:46:43.445656 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:46:43.445630 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:46:43.663465 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:43.663429 2577 generic.go:356] "Generic (PLEG): container finished" podID="23200d03-2bbb-49a2-a889-21ef6ef741de" containerID="039152b271bc4a87221896678963654b3cee2bfe3f64310d373def4adfbf6b9b" exitCode=6 Jul 16 21:46:43.663465 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:43.663471 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" event={"ID":"23200d03-2bbb-49a2-a889-21ef6ef741de","Type":"ContainerDied","Data":"039152b271bc4a87221896678963654b3cee2bfe3f64310d373def4adfbf6b9b"} Jul 16 21:46:43.663688 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:43.663526 2577 scope.go:117] "RemoveContainer" containerID="1490056ff57b50b51df985f18ed272c2eff9431a50c22ae6b11b90abc22dff27" Jul 16 21:46:43.663789 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:43.663771 2577 scope.go:117] "RemoveContainer" containerID="039152b271bc4a87221896678963654b3cee2bfe3f64310d373def4adfbf6b9b" Jul 16 21:46:43.664112 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:46:43.664089 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj_odh-ai-gateway-infra(23200d03-2bbb-49a2-a889-21ef6ef741de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" podUID="23200d03-2bbb-49a2-a889-21ef6ef741de" Jul 16 21:46:44.693009 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.692979 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj"] Jul 16 21:46:44.791565 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.791533 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6"] Jul 16 21:46:44.791915 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.791903 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f067acf8-5e20-4e5e-aa97-1011a3c7717e" containerName="revoke-keys" Jul 16 21:46:44.791955 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.791917 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f067acf8-5e20-4e5e-aa97-1011a3c7717e" containerName="revoke-keys" Jul 16 21:46:44.791955 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.791935 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f067acf8-5e20-4e5e-aa97-1011a3c7717e" containerName="revoke-keys" Jul 16 21:46:44.791955 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.791940 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f067acf8-5e20-4e5e-aa97-1011a3c7717e" containerName="revoke-keys" Jul 16 21:46:44.792066 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.791999 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f067acf8-5e20-4e5e-aa97-1011a3c7717e" containerName="revoke-keys" Jul 16 21:46:44.792066 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.792008 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f067acf8-5e20-4e5e-aa97-1011a3c7717e" containerName="revoke-keys" Jul 16 21:46:44.792066 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.792016 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f067acf8-5e20-4e5e-aa97-1011a3c7717e" containerName="revoke-keys" Jul 16 21:46:44.796517 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.796502 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" Jul 16 21:46:44.803516 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.803490 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6"] Jul 16 21:46:44.823489 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.823467 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" Jul 16 21:46:44.887504 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.887479 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b1f9ad-4bf8-4f32-9d98-15f29dcc8880-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6\" (UID: \"74b1f9ad-4bf8-4f32-9d98-15f29dcc8880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" Jul 16 21:46:44.988406 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.988380 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23200d03-2bbb-49a2-a889-21ef6ef741de-maas-api-service-ca\") pod \"23200d03-2bbb-49a2-a889-21ef6ef741de\" (UID: \"23200d03-2bbb-49a2-a889-21ef6ef741de\") " Jul 16 21:46:44.988540 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.988474 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b1f9ad-4bf8-4f32-9d98-15f29dcc8880-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6\" (UID: \"74b1f9ad-4bf8-4f32-9d98-15f29dcc8880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" Jul 16 21:46:44.988708 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.988687 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23200d03-2bbb-49a2-a889-21ef6ef741de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23200d03-2bbb-49a2-a889-21ef6ef741de" (UID: "23200d03-2bbb-49a2-a889-21ef6ef741de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:46:44.989081 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:44.989064 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b1f9ad-4bf8-4f32-9d98-15f29dcc8880-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6\" (UID: \"74b1f9ad-4bf8-4f32-9d98-15f29dcc8880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" Jul 16 21:46:45.089369 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:45.089337 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23200d03-2bbb-49a2-a889-21ef6ef741de-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:46:45.109203 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:45.109156 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" Jul 16 21:46:45.239624 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:45.239599 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6"] Jul 16 21:46:45.241075 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:46:45.241050 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b1f9ad_4bf8_4f32_9d98_15f29dcc8880.slice/crio-36486f4007b6a9e194ceb2a5e4955edb2276304b2e1536ec4eab132d417cf18e WatchSource:0}: Error finding container 36486f4007b6a9e194ceb2a5e4955edb2276304b2e1536ec4eab132d417cf18e: Status 404 returned error can't find the container with id 36486f4007b6a9e194ceb2a5e4955edb2276304b2e1536ec4eab132d417cf18e Jul 16 21:46:45.676556 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:45.676513 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" event={"ID":"74b1f9ad-4bf8-4f32-9d98-15f29dcc8880","Type":"ContainerStarted","Data":"6b1317919f966bd2b8c7d3429384a51eba480a4679571c6f300c47a4f4ee3624"} Jul 16 21:46:45.676739 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:45.676577 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" event={"ID":"74b1f9ad-4bf8-4f32-9d98-15f29dcc8880","Type":"ContainerStarted","Data":"36486f4007b6a9e194ceb2a5e4955edb2276304b2e1536ec4eab132d417cf18e"} Jul 16 21:46:45.677828 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:45.677812 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" Jul 16 21:46:45.677924 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:45.677853 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj" event={"ID":"23200d03-2bbb-49a2-a889-21ef6ef741de","Type":"ContainerDied","Data":"6615b4976c557b3037f47ed92335d5ec07056f7b0c6af0f376b051a13e805ba2"} Jul 16 21:46:45.677924 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:45.677897 2577 scope.go:117] "RemoveContainer" containerID="039152b271bc4a87221896678963654b3cee2bfe3f64310d373def4adfbf6b9b" Jul 16 21:46:45.694905 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:45.694863 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" podStartSLOduration=1.694849542 podStartE2EDuration="1.694849542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:46:45.692439727 +0000 UTC m=+2221.867547921" watchObservedRunningTime="2026-07-16 21:46:45.694849542 +0000 UTC m=+2221.869957783" Jul 16 21:46:45.707041 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:45.707014 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj"] Jul 16 21:46:45.712635 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:45.712614 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-lzbpj"] Jul 16 21:46:46.450132 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:46.450098 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23200d03-2bbb-49a2-a889-21ef6ef741de" path="/var/lib/kubelet/pods/23200d03-2bbb-49a2-a889-21ef6ef741de/volumes" Jul 16 21:46:46.688940 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:46.688904 2577 generic.go:356] "Generic (PLEG): container finished" podID="74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" containerID="6b1317919f966bd2b8c7d3429384a51eba480a4679571c6f300c47a4f4ee3624" exitCode=7 Jul 16 21:46:46.689102 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:46.688945 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" event={"ID":"74b1f9ad-4bf8-4f32-9d98-15f29dcc8880","Type":"ContainerDied","Data":"6b1317919f966bd2b8c7d3429384a51eba480a4679571c6f300c47a4f4ee3624"} Jul 16 21:46:46.689174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:46.689161 2577 scope.go:117] "RemoveContainer" containerID="6b1317919f966bd2b8c7d3429384a51eba480a4679571c6f300c47a4f4ee3624" Jul 16 21:46:47.701666 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:47.701630 2577 generic.go:356] "Generic (PLEG): container finished" podID="74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" containerID="839b60803fc777f225331999531d4252f5e22eb1874703c2883b31e004e960cf" exitCode=7 Jul 16 21:46:47.702121 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:47.701713 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" event={"ID":"74b1f9ad-4bf8-4f32-9d98-15f29dcc8880","Type":"ContainerDied","Data":"839b60803fc777f225331999531d4252f5e22eb1874703c2883b31e004e960cf"} Jul 16 21:46:47.702121 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:47.701787 2577 scope.go:117] "RemoveContainer" containerID="6b1317919f966bd2b8c7d3429384a51eba480a4679571c6f300c47a4f4ee3624" Jul 16 21:46:47.702121 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:47.701874 2577 scope.go:117] "RemoveContainer" containerID="839b60803fc777f225331999531d4252f5e22eb1874703c2883b31e004e960cf" Jul 16 21:46:47.702121 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:46:47.702069 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6_odh-ai-gateway-infra(74b1f9ad-4bf8-4f32-9d98-15f29dcc8880)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" podUID="74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" Jul 16 21:46:48.708422 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:48.708397 2577 scope.go:117] "RemoveContainer" containerID="839b60803fc777f225331999531d4252f5e22eb1874703c2883b31e004e960cf" Jul 16 21:46:48.708773 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:46:48.708586 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6_odh-ai-gateway-infra(74b1f9ad-4bf8-4f32-9d98-15f29dcc8880)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" podUID="74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" Jul 16 21:46:57.445498 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:46:57.445461 2577 scope.go:117] "RemoveContainer" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" Jul 16 21:46:57.445953 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:46:57.445672 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:47:00.444931 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:00.444901 2577 scope.go:117] "RemoveContainer" containerID="839b60803fc777f225331999531d4252f5e22eb1874703c2883b31e004e960cf" Jul 16 21:47:00.768887 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:00.768815 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" event={"ID":"74b1f9ad-4bf8-4f32-9d98-15f29dcc8880","Type":"ContainerStarted","Data":"ae534e391670657dbe231305f29d7bd6fbdc7994399ff857192b17263adf14cd"} Jul 16 21:47:01.775782 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:01.775747 2577 generic.go:356] "Generic (PLEG): container finished" podID="74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" containerID="ae534e391670657dbe231305f29d7bd6fbdc7994399ff857192b17263adf14cd" exitCode=7 Jul 16 21:47:01.776240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:01.775800 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" event={"ID":"74b1f9ad-4bf8-4f32-9d98-15f29dcc8880","Type":"ContainerDied","Data":"ae534e391670657dbe231305f29d7bd6fbdc7994399ff857192b17263adf14cd"} Jul 16 21:47:01.776240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:01.775848 2577 scope.go:117] "RemoveContainer" containerID="839b60803fc777f225331999531d4252f5e22eb1874703c2883b31e004e960cf" Jul 16 21:47:01.776240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:01.776080 2577 scope.go:117] "RemoveContainer" containerID="ae534e391670657dbe231305f29d7bd6fbdc7994399ff857192b17263adf14cd" Jul 16 21:47:01.776408 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:01.776310 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6_odh-ai-gateway-infra(74b1f9ad-4bf8-4f32-9d98-15f29dcc8880)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" podUID="74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" Jul 16 21:47:01.803791 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:01.803770 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6"] Jul 16 21:47:02.914124 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:02.914100 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" Jul 16 21:47:02.947165 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:02.947140 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b1f9ad-4bf8-4f32-9d98-15f29dcc8880-maas-api-service-ca\") pod \"74b1f9ad-4bf8-4f32-9d98-15f29dcc8880\" (UID: \"74b1f9ad-4bf8-4f32-9d98-15f29dcc8880\") " Jul 16 21:47:02.947504 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:02.947485 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b1f9ad-4bf8-4f32-9d98-15f29dcc8880-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" (UID: "74b1f9ad-4bf8-4f32-9d98-15f29dcc8880"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:47:03.047830 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:03.047748 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b1f9ad-4bf8-4f32-9d98-15f29dcc8880-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:47:03.788972 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:03.788943 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" Jul 16 21:47:03.789171 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:03.788995 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6" event={"ID":"74b1f9ad-4bf8-4f32-9d98-15f29dcc8880","Type":"ContainerDied","Data":"36486f4007b6a9e194ceb2a5e4955edb2276304b2e1536ec4eab132d417cf18e"} Jul 16 21:47:03.789171 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:03.789059 2577 scope.go:117] "RemoveContainer" containerID="ae534e391670657dbe231305f29d7bd6fbdc7994399ff857192b17263adf14cd" Jul 16 21:47:03.811394 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:03.811370 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6"] Jul 16 21:47:03.819055 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:03.819027 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-mjbc6"] Jul 16 21:47:04.451116 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:04.451083 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" path="/var/lib/kubelet/pods/74b1f9ad-4bf8-4f32-9d98-15f29dcc8880/volumes" Jul 16 21:47:05.453125 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453089 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d"] Jul 16 21:47:05.453525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453450 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" containerName="revoke-keys" Jul 16 21:47:05.453525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453463 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" containerName="revoke-keys" Jul 16 21:47:05.453525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453478 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23200d03-2bbb-49a2-a889-21ef6ef741de" containerName="revoke-keys" Jul 16 21:47:05.453525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453483 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23200d03-2bbb-49a2-a889-21ef6ef741de" containerName="revoke-keys" Jul 16 21:47:05.453525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453491 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23200d03-2bbb-49a2-a889-21ef6ef741de" containerName="revoke-keys" Jul 16 21:47:05.453525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453497 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23200d03-2bbb-49a2-a889-21ef6ef741de" containerName="revoke-keys" Jul 16 21:47:05.453525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453506 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" containerName="revoke-keys" Jul 16 21:47:05.453525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453511 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" containerName="revoke-keys" Jul 16 21:47:05.453525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453518 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" containerName="revoke-keys" Jul 16 21:47:05.453525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453523 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" containerName="revoke-keys" Jul 16 21:47:05.453874 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453532 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f067acf8-5e20-4e5e-aa97-1011a3c7717e" containerName="revoke-keys" Jul 16 21:47:05.453874 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453537 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f067acf8-5e20-4e5e-aa97-1011a3c7717e" containerName="revoke-keys" Jul 16 21:47:05.453874 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453588 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="23200d03-2bbb-49a2-a889-21ef6ef741de" containerName="revoke-keys" Jul 16 21:47:05.453874 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453595 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" containerName="revoke-keys" Jul 16 21:47:05.453874 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453601 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" containerName="revoke-keys" Jul 16 21:47:05.453874 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453609 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="23200d03-2bbb-49a2-a889-21ef6ef741de" containerName="revoke-keys" Jul 16 21:47:05.453874 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.453615 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="23200d03-2bbb-49a2-a889-21ef6ef741de" containerName="revoke-keys" Jul 16 21:47:05.457998 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.457981 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" Jul 16 21:47:05.460723 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.460706 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:47:05.464926 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.464882 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d"] Jul 16 21:47:05.465055 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.464944 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1649e6d6-7f61-4478-81d9-1673ffb096f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d\" (UID: \"1649e6d6-7f61-4478-81d9-1673ffb096f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" Jul 16 21:47:05.566431 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.566403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1649e6d6-7f61-4478-81d9-1673ffb096f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d\" (UID: \"1649e6d6-7f61-4478-81d9-1673ffb096f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" Jul 16 21:47:05.566991 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.566975 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1649e6d6-7f61-4478-81d9-1673ffb096f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d\" (UID: \"1649e6d6-7f61-4478-81d9-1673ffb096f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" Jul 16 21:47:05.770278 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.770155 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" Jul 16 21:47:05.903023 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:05.903002 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d"] Jul 16 21:47:05.904708 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:47:05.904683 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1649e6d6_7f61_4478_81d9_1673ffb096f9.slice/crio-3ada96234872e4344a8955de4c7d2830a68b005b51b5beb7f9f5fafd289c9a50 WatchSource:0}: Error finding container 3ada96234872e4344a8955de4c7d2830a68b005b51b5beb7f9f5fafd289c9a50: Status 404 returned error can't find the container with id 3ada96234872e4344a8955de4c7d2830a68b005b51b5beb7f9f5fafd289c9a50 Jul 16 21:47:06.805677 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:06.805589 2577 generic.go:356] "Generic (PLEG): container finished" podID="1649e6d6-7f61-4478-81d9-1673ffb096f9" containerID="bb7c8176bb13f9dae3404ccacf9134f5e72fdefe8678af8eb2a56943c784fcf8" exitCode=6 Jul 16 21:47:06.806019 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:06.805680 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" event={"ID":"1649e6d6-7f61-4478-81d9-1673ffb096f9","Type":"ContainerDied","Data":"bb7c8176bb13f9dae3404ccacf9134f5e72fdefe8678af8eb2a56943c784fcf8"} Jul 16 21:47:06.806019 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:06.805731 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" event={"ID":"1649e6d6-7f61-4478-81d9-1673ffb096f9","Type":"ContainerStarted","Data":"3ada96234872e4344a8955de4c7d2830a68b005b51b5beb7f9f5fafd289c9a50"} Jul 16 21:47:06.806019 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:06.805877 2577 scope.go:117] "RemoveContainer" containerID="bb7c8176bb13f9dae3404ccacf9134f5e72fdefe8678af8eb2a56943c784fcf8" Jul 16 21:47:07.812495 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:07.812410 2577 generic.go:356] "Generic (PLEG): container finished" podID="1649e6d6-7f61-4478-81d9-1673ffb096f9" containerID="fd59dcc72336d4dae1feb9cb1f327752527cc0ecca89cf3f25b86dd0fd9833d3" exitCode=6 Jul 16 21:47:07.812886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:07.812496 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" event={"ID":"1649e6d6-7f61-4478-81d9-1673ffb096f9","Type":"ContainerDied","Data":"fd59dcc72336d4dae1feb9cb1f327752527cc0ecca89cf3f25b86dd0fd9833d3"} Jul 16 21:47:07.812886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:07.812553 2577 scope.go:117] "RemoveContainer" containerID="bb7c8176bb13f9dae3404ccacf9134f5e72fdefe8678af8eb2a56943c784fcf8" Jul 16 21:47:07.812886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:07.812773 2577 scope.go:117] "RemoveContainer" containerID="fd59dcc72336d4dae1feb9cb1f327752527cc0ecca89cf3f25b86dd0fd9833d3" Jul 16 21:47:07.813039 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:07.813018 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d_odh-ai-gateway-infra(1649e6d6-7f61-4478-81d9-1673ffb096f9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" podUID="1649e6d6-7f61-4478-81d9-1673ffb096f9" Jul 16 21:47:08.819911 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:08.819885 2577 scope.go:117] "RemoveContainer" containerID="fd59dcc72336d4dae1feb9cb1f327752527cc0ecca89cf3f25b86dd0fd9833d3" Jul 16 21:47:08.820333 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:08.820080 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d_odh-ai-gateway-infra(1649e6d6-7f61-4478-81d9-1673ffb096f9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" podUID="1649e6d6-7f61-4478-81d9-1673ffb096f9" Jul 16 21:47:09.445224 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:09.445169 2577 scope.go:117] "RemoveContainer" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" Jul 16 21:47:09.445405 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:09.445376 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:47:16.599201 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:16.599157 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2"] Jul 16 21:47:16.599586 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:16.599509 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23200d03-2bbb-49a2-a889-21ef6ef741de" containerName="revoke-keys" Jul 16 21:47:16.599586 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:16.599523 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23200d03-2bbb-49a2-a889-21ef6ef741de" containerName="revoke-keys" Jul 16 21:47:16.599586 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:16.599577 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74b1f9ad-4bf8-4f32-9d98-15f29dcc8880" containerName="revoke-keys" Jul 16 21:47:16.603855 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:16.603841 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" Jul 16 21:47:16.612321 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:16.612298 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2"] Jul 16 21:47:16.660230 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:16.660172 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e26870-4b85-45a1-a160-a458a14a9208-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2\" (UID: \"e9e26870-4b85-45a1-a160-a458a14a9208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" Jul 16 21:47:16.761038 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:16.760994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e26870-4b85-45a1-a160-a458a14a9208-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2\" (UID: \"e9e26870-4b85-45a1-a160-a458a14a9208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" Jul 16 21:47:16.761630 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:16.761612 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e26870-4b85-45a1-a160-a458a14a9208-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2\" (UID: \"e9e26870-4b85-45a1-a160-a458a14a9208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" Jul 16 21:47:16.915416 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:16.915388 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" Jul 16 21:47:17.064302 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:17.064264 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2"] Jul 16 21:47:17.067453 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:47:17.067423 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e26870_4b85_45a1_a160_a458a14a9208.slice/crio-254b667ab2cbcc8d314c39f39ef6ced16e78a4c1514a5cc79f7bd87e57d22683 WatchSource:0}: Error finding container 254b667ab2cbcc8d314c39f39ef6ced16e78a4c1514a5cc79f7bd87e57d22683: Status 404 returned error can't find the container with id 254b667ab2cbcc8d314c39f39ef6ced16e78a4c1514a5cc79f7bd87e57d22683 Jul 16 21:47:17.864968 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:17.864933 2577 generic.go:356] "Generic (PLEG): container finished" podID="e9e26870-4b85-45a1-a160-a458a14a9208" containerID="add1754c9de9addbb37341651da680e766daecd64bc67531aa06d2803137a17d" exitCode=6 Jul 16 21:47:17.865355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:17.865019 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" event={"ID":"e9e26870-4b85-45a1-a160-a458a14a9208","Type":"ContainerDied","Data":"add1754c9de9addbb37341651da680e766daecd64bc67531aa06d2803137a17d"} Jul 16 21:47:17.865355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:17.865114 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" event={"ID":"e9e26870-4b85-45a1-a160-a458a14a9208","Type":"ContainerStarted","Data":"254b667ab2cbcc8d314c39f39ef6ced16e78a4c1514a5cc79f7bd87e57d22683"} Jul 16 21:47:17.865355 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:17.865205 2577 scope.go:117] "RemoveContainer" containerID="add1754c9de9addbb37341651da680e766daecd64bc67531aa06d2803137a17d" Jul 16 21:47:18.872057 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:18.871960 2577 generic.go:356] "Generic (PLEG): container finished" podID="e9e26870-4b85-45a1-a160-a458a14a9208" containerID="dd6114c6818b0e36c62740c846157065b7999f3bada2174cd4baed2017789e5f" exitCode=6 Jul 16 21:47:18.872057 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:18.872046 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" event={"ID":"e9e26870-4b85-45a1-a160-a458a14a9208","Type":"ContainerDied","Data":"dd6114c6818b0e36c62740c846157065b7999f3bada2174cd4baed2017789e5f"} Jul 16 21:47:18.872525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:18.872100 2577 scope.go:117] "RemoveContainer" containerID="add1754c9de9addbb37341651da680e766daecd64bc67531aa06d2803137a17d" Jul 16 21:47:18.872525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:18.872238 2577 scope.go:117] "RemoveContainer" containerID="dd6114c6818b0e36c62740c846157065b7999f3bada2174cd4baed2017789e5f" Jul 16 21:47:18.872525 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:18.872498 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2_odh-ai-gateway-infra(e9e26870-4b85-45a1-a160-a458a14a9208)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" podUID="e9e26870-4b85-45a1-a160-a458a14a9208" Jul 16 21:47:19.879448 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:19.879422 2577 scope.go:117] "RemoveContainer" containerID="dd6114c6818b0e36c62740c846157065b7999f3bada2174cd4baed2017789e5f" Jul 16 21:47:19.879805 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:19.879598 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2_odh-ai-gateway-infra(e9e26870-4b85-45a1-a160-a458a14a9208)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" podUID="e9e26870-4b85-45a1-a160-a458a14a9208" Jul 16 21:47:20.445242 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:20.445217 2577 scope.go:117] "RemoveContainer" containerID="fd59dcc72336d4dae1feb9cb1f327752527cc0ecca89cf3f25b86dd0fd9833d3" Jul 16 21:47:20.885795 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:20.885767 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" event={"ID":"1649e6d6-7f61-4478-81d9-1673ffb096f9","Type":"ContainerStarted","Data":"4c810dfbbed7c34c540dddeff2162eefae65993273aa4259f9f2f206ae24d2a3"} Jul 16 21:47:20.903723 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:20.903676 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" podStartSLOduration=15.903661851 podStartE2EDuration="15.903661851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:20.903067049 +0000 UTC m=+2257.078175279" watchObservedRunningTime="2026-07-16 21:47:20.903661851 +0000 UTC m=+2257.078770043" Jul 16 21:47:21.892803 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:21.892767 2577 generic.go:356] "Generic (PLEG): container finished" podID="1649e6d6-7f61-4478-81d9-1673ffb096f9" containerID="4c810dfbbed7c34c540dddeff2162eefae65993273aa4259f9f2f206ae24d2a3" exitCode=6 Jul 16 21:47:21.893215 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:21.892837 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" event={"ID":"1649e6d6-7f61-4478-81d9-1673ffb096f9","Type":"ContainerDied","Data":"4c810dfbbed7c34c540dddeff2162eefae65993273aa4259f9f2f206ae24d2a3"} Jul 16 21:47:21.893215 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:21.892901 2577 scope.go:117] "RemoveContainer" containerID="fd59dcc72336d4dae1feb9cb1f327752527cc0ecca89cf3f25b86dd0fd9833d3" Jul 16 21:47:21.893349 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:21.893240 2577 scope.go:117] "RemoveContainer" containerID="4c810dfbbed7c34c540dddeff2162eefae65993273aa4259f9f2f206ae24d2a3" Jul 16 21:47:21.893460 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:21.893435 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d_odh-ai-gateway-infra(1649e6d6-7f61-4478-81d9-1673ffb096f9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" podUID="1649e6d6-7f61-4478-81d9-1673ffb096f9" Jul 16 21:47:21.919685 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:21.919658 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d"] Jul 16 21:47:22.445757 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:22.445725 2577 scope.go:117] "RemoveContainer" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" Jul 16 21:47:22.445971 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:22.445946 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:47:23.026680 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:23.026656 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" Jul 16 21:47:23.115336 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:23.115310 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1649e6d6-7f61-4478-81d9-1673ffb096f9-maas-api-service-ca\") pod \"1649e6d6-7f61-4478-81d9-1673ffb096f9\" (UID: \"1649e6d6-7f61-4478-81d9-1673ffb096f9\") " Jul 16 21:47:23.115665 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:23.115641 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1649e6d6-7f61-4478-81d9-1673ffb096f9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1649e6d6-7f61-4478-81d9-1673ffb096f9" (UID: "1649e6d6-7f61-4478-81d9-1673ffb096f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:47:23.216407 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:23.216323 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1649e6d6-7f61-4478-81d9-1673ffb096f9-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:47:23.906979 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:23.906944 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" event={"ID":"1649e6d6-7f61-4478-81d9-1673ffb096f9","Type":"ContainerDied","Data":"3ada96234872e4344a8955de4c7d2830a68b005b51b5beb7f9f5fafd289c9a50"} Jul 16 21:47:23.907134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:23.907008 2577 scope.go:117] "RemoveContainer" containerID="4c810dfbbed7c34c540dddeff2162eefae65993273aa4259f9f2f206ae24d2a3" Jul 16 21:47:23.907134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:23.906953 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d" Jul 16 21:47:23.933448 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:23.933422 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d"] Jul 16 21:47:23.939573 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:23.939549 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-mfd2d"] Jul 16 21:47:24.450270 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:24.450233 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1649e6d6-7f61-4478-81d9-1673ffb096f9" path="/var/lib/kubelet/pods/1649e6d6-7f61-4478-81d9-1673ffb096f9/volumes" Jul 16 21:47:31.445367 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:31.445337 2577 scope.go:117] "RemoveContainer" containerID="dd6114c6818b0e36c62740c846157065b7999f3bada2174cd4baed2017789e5f" Jul 16 21:47:31.949646 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:31.949613 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" event={"ID":"e9e26870-4b85-45a1-a160-a458a14a9208","Type":"ContainerStarted","Data":"835a49a8f27b94bfa056c275eefb3a2c6675a48e613b99a78cbda79f8c85bcd5"} Jul 16 21:47:31.969352 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:31.969303 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" podStartSLOduration=15.969286731 podStartE2EDuration="15.969286731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:31.967917261 +0000 UTC m=+2268.143025478" watchObservedRunningTime="2026-07-16 21:47:31.969286731 +0000 UTC m=+2268.144394960" Jul 16 21:47:32.957088 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:32.957045 2577 generic.go:356] "Generic (PLEG): container finished" podID="e9e26870-4b85-45a1-a160-a458a14a9208" containerID="835a49a8f27b94bfa056c275eefb3a2c6675a48e613b99a78cbda79f8c85bcd5" exitCode=6 Jul 16 21:47:32.957592 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:32.957118 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" event={"ID":"e9e26870-4b85-45a1-a160-a458a14a9208","Type":"ContainerDied","Data":"835a49a8f27b94bfa056c275eefb3a2c6675a48e613b99a78cbda79f8c85bcd5"} Jul 16 21:47:32.957592 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:32.957177 2577 scope.go:117] "RemoveContainer" containerID="dd6114c6818b0e36c62740c846157065b7999f3bada2174cd4baed2017789e5f" Jul 16 21:47:32.957592 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:32.957524 2577 scope.go:117] "RemoveContainer" containerID="835a49a8f27b94bfa056c275eefb3a2c6675a48e613b99a78cbda79f8c85bcd5" Jul 16 21:47:32.957798 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:32.957777 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2_odh-ai-gateway-infra(e9e26870-4b85-45a1-a160-a458a14a9208)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" podUID="e9e26870-4b85-45a1-a160-a458a14a9208" Jul 16 21:47:32.983928 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:32.983906 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2"] Jul 16 21:47:34.114766 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.114745 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" Jul 16 21:47:34.209687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.209651 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e26870-4b85-45a1-a160-a458a14a9208-maas-api-service-ca\") pod \"e9e26870-4b85-45a1-a160-a458a14a9208\" (UID: \"e9e26870-4b85-45a1-a160-a458a14a9208\") " Jul 16 21:47:34.210061 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.210042 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e26870-4b85-45a1-a160-a458a14a9208-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9e26870-4b85-45a1-a160-a458a14a9208" (UID: "e9e26870-4b85-45a1-a160-a458a14a9208"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:47:34.311003 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.310914 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e26870-4b85-45a1-a160-a458a14a9208-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:47:34.834705 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.834674 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8"] Jul 16 21:47:34.835022 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835009 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e26870-4b85-45a1-a160-a458a14a9208" containerName="revoke-keys" Jul 16 21:47:34.835073 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835023 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e26870-4b85-45a1-a160-a458a14a9208" containerName="revoke-keys" Jul 16 21:47:34.835073 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835032 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1649e6d6-7f61-4478-81d9-1673ffb096f9" containerName="revoke-keys" Jul 16 21:47:34.835073 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835039 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1649e6d6-7f61-4478-81d9-1673ffb096f9" containerName="revoke-keys" Jul 16 21:47:34.835073 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835050 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1649e6d6-7f61-4478-81d9-1673ffb096f9" containerName="revoke-keys" Jul 16 21:47:34.835073 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835055 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1649e6d6-7f61-4478-81d9-1673ffb096f9" containerName="revoke-keys" Jul 16 21:47:34.835073 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835064 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e26870-4b85-45a1-a160-a458a14a9208" containerName="revoke-keys" Jul 16 21:47:34.835073 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835069 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e26870-4b85-45a1-a160-a458a14a9208" containerName="revoke-keys" Jul 16 21:47:34.835073 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835074 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e26870-4b85-45a1-a160-a458a14a9208" containerName="revoke-keys" Jul 16 21:47:34.835338 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835080 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e26870-4b85-45a1-a160-a458a14a9208" containerName="revoke-keys" Jul 16 21:47:34.835338 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835086 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1649e6d6-7f61-4478-81d9-1673ffb096f9" containerName="revoke-keys" Jul 16 21:47:34.835338 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835091 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1649e6d6-7f61-4478-81d9-1673ffb096f9" containerName="revoke-keys" Jul 16 21:47:34.835338 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835146 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9e26870-4b85-45a1-a160-a458a14a9208" containerName="revoke-keys" Jul 16 21:47:34.835338 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835152 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1649e6d6-7f61-4478-81d9-1673ffb096f9" containerName="revoke-keys" Jul 16 21:47:34.835338 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835159 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9e26870-4b85-45a1-a160-a458a14a9208" containerName="revoke-keys" Jul 16 21:47:34.835338 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835165 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1649e6d6-7f61-4478-81d9-1673ffb096f9" containerName="revoke-keys" Jul 16 21:47:34.835338 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835171 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1649e6d6-7f61-4478-81d9-1673ffb096f9" containerName="revoke-keys" Jul 16 21:47:34.835338 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.835177 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9e26870-4b85-45a1-a160-a458a14a9208" containerName="revoke-keys" Jul 16 21:47:34.838214 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.838179 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" Jul 16 21:47:34.846510 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.846486 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8"] Jul 16 21:47:34.915852 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.915818 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45d89610-cc1b-43f1-8d0a-410e1da2829b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8\" (UID: \"45d89610-cc1b-43f1-8d0a-410e1da2829b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" Jul 16 21:47:34.969390 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.969358 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" Jul 16 21:47:34.969550 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.969394 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2" event={"ID":"e9e26870-4b85-45a1-a160-a458a14a9208","Type":"ContainerDied","Data":"254b667ab2cbcc8d314c39f39ef6ced16e78a4c1514a5cc79f7bd87e57d22683"} Jul 16 21:47:34.969550 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.969456 2577 scope.go:117] "RemoveContainer" containerID="835a49a8f27b94bfa056c275eefb3a2c6675a48e613b99a78cbda79f8c85bcd5" Jul 16 21:47:34.988771 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.988746 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2"] Jul 16 21:47:34.993746 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:34.993725 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-xqgl2"] Jul 16 21:47:35.017052 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:35.017032 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45d89610-cc1b-43f1-8d0a-410e1da2829b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8\" (UID: \"45d89610-cc1b-43f1-8d0a-410e1da2829b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" Jul 16 21:47:35.017659 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:35.017641 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45d89610-cc1b-43f1-8d0a-410e1da2829b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8\" (UID: \"45d89610-cc1b-43f1-8d0a-410e1da2829b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" Jul 16 21:47:35.149066 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:35.149033 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" Jul 16 21:47:35.283597 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:35.283570 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8"] Jul 16 21:47:35.284797 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:47:35.284771 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d89610_cc1b_43f1_8d0a_410e1da2829b.slice/crio-ea468b0c281ff869827915b02caa0ab5ce45c884d45d2a515912371958f9996d WatchSource:0}: Error finding container ea468b0c281ff869827915b02caa0ab5ce45c884d45d2a515912371958f9996d: Status 404 returned error can't find the container with id ea468b0c281ff869827915b02caa0ab5ce45c884d45d2a515912371958f9996d Jul 16 21:47:35.977485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:35.977454 2577 generic.go:356] "Generic (PLEG): container finished" podID="45d89610-cc1b-43f1-8d0a-410e1da2829b" containerID="b4e50444c5d7c35dec41775396166514c8ba36028dfb21185958abc16cd879d1" exitCode=7 Jul 16 21:47:35.977640 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:35.977548 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" event={"ID":"45d89610-cc1b-43f1-8d0a-410e1da2829b","Type":"ContainerDied","Data":"b4e50444c5d7c35dec41775396166514c8ba36028dfb21185958abc16cd879d1"} Jul 16 21:47:35.977640 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:35.977607 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" event={"ID":"45d89610-cc1b-43f1-8d0a-410e1da2829b","Type":"ContainerStarted","Data":"ea468b0c281ff869827915b02caa0ab5ce45c884d45d2a515912371958f9996d"} Jul 16 21:47:35.977736 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:35.977724 2577 scope.go:117] "RemoveContainer" containerID="b4e50444c5d7c35dec41775396166514c8ba36028dfb21185958abc16cd879d1" Jul 16 21:47:36.451246 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:36.451156 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9e26870-4b85-45a1-a160-a458a14a9208" path="/var/lib/kubelet/pods/e9e26870-4b85-45a1-a160-a458a14a9208/volumes" Jul 16 21:47:36.984777 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:36.984745 2577 generic.go:356] "Generic (PLEG): container finished" podID="45d89610-cc1b-43f1-8d0a-410e1da2829b" containerID="4674060f4da97b9cdbbee8478832d59b0bb20da9b17bd2723cc210cda3973b02" exitCode=7 Jul 16 21:47:36.984950 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:36.984808 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" event={"ID":"45d89610-cc1b-43f1-8d0a-410e1da2829b","Type":"ContainerDied","Data":"4674060f4da97b9cdbbee8478832d59b0bb20da9b17bd2723cc210cda3973b02"} Jul 16 21:47:36.984950 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:36.984851 2577 scope.go:117] "RemoveContainer" containerID="b4e50444c5d7c35dec41775396166514c8ba36028dfb21185958abc16cd879d1" Jul 16 21:47:36.985102 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:36.985083 2577 scope.go:117] "RemoveContainer" containerID="4674060f4da97b9cdbbee8478832d59b0bb20da9b17bd2723cc210cda3973b02" Jul 16 21:47:36.985337 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:36.985317 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8_odh-ai-gateway-infra(45d89610-cc1b-43f1-8d0a-410e1da2829b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" podUID="45d89610-cc1b-43f1-8d0a-410e1da2829b" Jul 16 21:47:37.444928 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:37.444899 2577 scope.go:117] "RemoveContainer" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" Jul 16 21:47:37.445093 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:37.445071 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:47:37.995065 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:37.995039 2577 scope.go:117] "RemoveContainer" containerID="4674060f4da97b9cdbbee8478832d59b0bb20da9b17bd2723cc210cda3973b02" Jul 16 21:47:37.995470 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:37.995248 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8_odh-ai-gateway-infra(45d89610-cc1b-43f1-8d0a-410e1da2829b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" podUID="45d89610-cc1b-43f1-8d0a-410e1da2829b" Jul 16 21:47:48.450123 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:48.450093 2577 scope.go:117] "RemoveContainer" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" Jul 16 21:47:48.450558 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:48.450228 2577 scope.go:117] "RemoveContainer" containerID="4674060f4da97b9cdbbee8478832d59b0bb20da9b17bd2723cc210cda3973b02" Jul 16 21:47:49.059670 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:49.059647 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" event={"ID":"45d89610-cc1b-43f1-8d0a-410e1da2829b","Type":"ContainerStarted","Data":"0f29d78419d722921476563f85e667589caece69925c515641247a7f1061336c"} Jul 16 21:47:49.079052 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:49.079008 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" podStartSLOduration=15.078993426 podStartE2EDuration="15.078993426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:49.077909407 +0000 UTC m=+2285.253017600" watchObservedRunningTime="2026-07-16 21:47:49.078993426 +0000 UTC m=+2285.254101617" Jul 16 21:47:50.066526 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:50.066489 2577 generic.go:356] "Generic (PLEG): container finished" podID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerID="085992d52f8f3be350529602b7ff0fcb1bb0ec6236b05dbde6ccbf46463b3029" exitCode=1 Jul 16 21:47:50.066996 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:50.066558 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" event={"ID":"454a4f80-c2dd-45b4-9b59-9152db8367d1","Type":"ContainerDied","Data":"085992d52f8f3be350529602b7ff0fcb1bb0ec6236b05dbde6ccbf46463b3029"} Jul 16 21:47:50.066996 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:50.066619 2577 scope.go:117] "RemoveContainer" containerID="fa3f416738c2dc9aebe01959181349a37e96db22876448a488750f8dd9c9e249" Jul 16 21:47:50.066996 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:50.066957 2577 scope.go:117] "RemoveContainer" containerID="085992d52f8f3be350529602b7ff0fcb1bb0ec6236b05dbde6ccbf46463b3029" Jul 16 21:47:50.067205 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:50.067166 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:47:50.068387 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:50.068367 2577 generic.go:356] "Generic (PLEG): container finished" podID="45d89610-cc1b-43f1-8d0a-410e1da2829b" containerID="0f29d78419d722921476563f85e667589caece69925c515641247a7f1061336c" exitCode=7 Jul 16 21:47:50.068485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:50.068429 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" event={"ID":"45d89610-cc1b-43f1-8d0a-410e1da2829b","Type":"ContainerDied","Data":"0f29d78419d722921476563f85e667589caece69925c515641247a7f1061336c"} Jul 16 21:47:50.068698 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:50.068686 2577 scope.go:117] "RemoveContainer" containerID="0f29d78419d722921476563f85e667589caece69925c515641247a7f1061336c" Jul 16 21:47:50.068840 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:50.068822 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8_odh-ai-gateway-infra(45d89610-cc1b-43f1-8d0a-410e1da2829b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" podUID="45d89610-cc1b-43f1-8d0a-410e1da2829b" Jul 16 21:47:50.077717 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:50.077701 2577 scope.go:117] "RemoveContainer" containerID="4674060f4da97b9cdbbee8478832d59b0bb20da9b17bd2723cc210cda3973b02" Jul 16 21:47:50.090597 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:50.090578 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8"] Jul 16 21:47:50.403877 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:50.403851 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:47:51.075737 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:51.075714 2577 scope.go:117] "RemoveContainer" containerID="085992d52f8f3be350529602b7ff0fcb1bb0ec6236b05dbde6ccbf46463b3029" Jul 16 21:47:51.076228 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:51.075938 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:47:51.209566 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:51.209545 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" Jul 16 21:47:51.354319 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:51.354247 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45d89610-cc1b-43f1-8d0a-410e1da2829b-maas-api-service-ca\") pod \"45d89610-cc1b-43f1-8d0a-410e1da2829b\" (UID: \"45d89610-cc1b-43f1-8d0a-410e1da2829b\") " Jul 16 21:47:51.354606 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:51.354588 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d89610-cc1b-43f1-8d0a-410e1da2829b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "45d89610-cc1b-43f1-8d0a-410e1da2829b" (UID: "45d89610-cc1b-43f1-8d0a-410e1da2829b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:47:51.454846 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:51.454815 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45d89610-cc1b-43f1-8d0a-410e1da2829b-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:47:52.083375 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:52.083343 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" Jul 16 21:47:52.083822 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:52.083332 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8" event={"ID":"45d89610-cc1b-43f1-8d0a-410e1da2829b","Type":"ContainerDied","Data":"ea468b0c281ff869827915b02caa0ab5ce45c884d45d2a515912371958f9996d"} Jul 16 21:47:52.083822 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:52.083546 2577 scope.go:117] "RemoveContainer" containerID="0f29d78419d722921476563f85e667589caece69925c515641247a7f1061336c" Jul 16 21:47:52.106195 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:52.106156 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8"] Jul 16 21:47:52.110359 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:52.110335 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sf7g8"] Jul 16 21:47:52.450640 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:52.450599 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45d89610-cc1b-43f1-8d0a-410e1da2829b" path="/var/lib/kubelet/pods/45d89610-cc1b-43f1-8d0a-410e1da2829b/volumes" Jul 16 21:47:55.006470 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.006433 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:47:55.006864 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.006769 2577 scope.go:117] "RemoveContainer" containerID="085992d52f8f3be350529602b7ff0fcb1bb0ec6236b05dbde6ccbf46463b3029" Jul 16 21:47:55.006952 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:55.006933 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:47:55.492363 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.492334 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb"] Jul 16 21:47:55.492692 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.492678 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d89610-cc1b-43f1-8d0a-410e1da2829b" containerName="revoke-keys" Jul 16 21:47:55.492692 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.492692 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d89610-cc1b-43f1-8d0a-410e1da2829b" containerName="revoke-keys" Jul 16 21:47:55.492819 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.492706 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d89610-cc1b-43f1-8d0a-410e1da2829b" containerName="revoke-keys" Jul 16 21:47:55.492819 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.492712 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d89610-cc1b-43f1-8d0a-410e1da2829b" containerName="revoke-keys" Jul 16 21:47:55.492819 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.492765 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d89610-cc1b-43f1-8d0a-410e1da2829b" containerName="revoke-keys" Jul 16 21:47:55.492819 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.492772 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d89610-cc1b-43f1-8d0a-410e1da2829b" containerName="revoke-keys" Jul 16 21:47:55.492819 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.492778 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d89610-cc1b-43f1-8d0a-410e1da2829b" containerName="revoke-keys" Jul 16 21:47:55.497058 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.497042 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb" Jul 16 21:47:55.500106 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.500085 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:47:55.504732 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.504706 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb"] Jul 16 21:47:55.590894 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.590862 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca3a37e-1f57-485a-9478-6f6539161f48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb\" (UID: \"2ca3a37e-1f57-485a-9478-6f6539161f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb" Jul 16 21:47:55.691419 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.691382 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca3a37e-1f57-485a-9478-6f6539161f48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb\" (UID: \"2ca3a37e-1f57-485a-9478-6f6539161f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb" Jul 16 21:47:55.691995 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.691977 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca3a37e-1f57-485a-9478-6f6539161f48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb\" (UID: \"2ca3a37e-1f57-485a-9478-6f6539161f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb" Jul 16 21:47:55.808701 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:55.808634 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb" Jul 16 21:47:56.140548 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:56.140525 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb"] Jul 16 21:47:56.142652 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:47:56.142625 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca3a37e_1f57_485a_9478_6f6539161f48.slice/crio-cc6c3400b892f45c4798582652dd48c4d97084dc786b65b13a551085d0c27657 WatchSource:0}: Error finding container cc6c3400b892f45c4798582652dd48c4d97084dc786b65b13a551085d0c27657: Status 404 returned error can't find the container with id cc6c3400b892f45c4798582652dd48c4d97084dc786b65b13a551085d0c27657 Jul 16 21:47:57.110108 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:57.110028 2577 generic.go:356] "Generic (PLEG): container finished" podID="2ca3a37e-1f57-485a-9478-6f6539161f48" containerID="cc2f3db27c0bfadfa79303c60c07057ed0c5db94e182d795e747548863a1ca33" exitCode=6 Jul 16 21:47:57.110108 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:57.110064 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb" event={"ID":"2ca3a37e-1f57-485a-9478-6f6539161f48","Type":"ContainerDied","Data":"cc2f3db27c0bfadfa79303c60c07057ed0c5db94e182d795e747548863a1ca33"} Jul 16 21:47:57.110108 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:57.110099 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb" event={"ID":"2ca3a37e-1f57-485a-9478-6f6539161f48","Type":"ContainerStarted","Data":"cc6c3400b892f45c4798582652dd48c4d97084dc786b65b13a551085d0c27657"} Jul 16 21:47:57.110356 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:57.110303 2577 scope.go:117] "RemoveContainer" containerID="cc2f3db27c0bfadfa79303c60c07057ed0c5db94e182d795e747548863a1ca33" Jul 16 21:47:58.122266 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:58.122164 2577 generic.go:356] "Generic (PLEG): container finished" podID="2ca3a37e-1f57-485a-9478-6f6539161f48" containerID="cf1a3dd5ac3ba33f8372370886f9cfd2a35e18c43fbfcc164fc126b1016a1c99" exitCode=6 Jul 16 21:47:58.122266 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:58.122237 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb" event={"ID":"2ca3a37e-1f57-485a-9478-6f6539161f48","Type":"ContainerDied","Data":"cf1a3dd5ac3ba33f8372370886f9cfd2a35e18c43fbfcc164fc126b1016a1c99"} Jul 16 21:47:58.122722 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:58.122333 2577 scope.go:117] "RemoveContainer" containerID="cc2f3db27c0bfadfa79303c60c07057ed0c5db94e182d795e747548863a1ca33" Jul 16 21:47:58.122722 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:58.122540 2577 scope.go:117] "RemoveContainer" containerID="cf1a3dd5ac3ba33f8372370886f9cfd2a35e18c43fbfcc164fc126b1016a1c99" Jul 16 21:47:58.122818 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:58.122720 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb_odh-ai-gateway-infra(2ca3a37e-1f57-485a-9478-6f6539161f48)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb" podUID="2ca3a37e-1f57-485a-9478-6f6539161f48" Jul 16 21:47:59.129734 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:47:59.129708 2577 scope.go:117] "RemoveContainer" containerID="cf1a3dd5ac3ba33f8372370886f9cfd2a35e18c43fbfcc164fc126b1016a1c99" Jul 16 21:47:59.130108 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:47:59.129929 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb_odh-ai-gateway-infra(2ca3a37e-1f57-485a-9478-6f6539161f48)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb" podUID="2ca3a37e-1f57-485a-9478-6f6539161f48" Jul 16 21:48:05.444969 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:05.444938 2577 scope.go:117] "RemoveContainer" containerID="085992d52f8f3be350529602b7ff0fcb1bb0ec6236b05dbde6ccbf46463b3029" Jul 16 21:48:05.445453 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:05.445116 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-f60157c3-54749c956b-dswbg_odh-ai-gateway-infra(454a4f80-c2dd-45b4-9b59-9152db8367d1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" Jul 16 21:48:06.635771 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:06.635734 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh"] Jul 16 21:48:06.636179 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:06.636093 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d89610-cc1b-43f1-8d0a-410e1da2829b" containerName="revoke-keys" Jul 16 21:48:06.636179 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:06.636106 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d89610-cc1b-43f1-8d0a-410e1da2829b" containerName="revoke-keys" Jul 16 21:48:06.640469 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:06.640453 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh" Jul 16 21:48:06.647074 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:06.647049 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh"] Jul 16 21:48:06.685709 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:06.685682 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7fe99c-9b73-4956-b173-095a8f504b3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh\" (UID: \"3f7fe99c-9b73-4956-b173-095a8f504b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh" Jul 16 21:48:06.786881 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:06.786853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7fe99c-9b73-4956-b173-095a8f504b3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh\" (UID: \"3f7fe99c-9b73-4956-b173-095a8f504b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh" Jul 16 21:48:06.787539 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:06.787518 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7fe99c-9b73-4956-b173-095a8f504b3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh\" (UID: \"3f7fe99c-9b73-4956-b173-095a8f504b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh" Jul 16 21:48:06.952164 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:06.952089 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh" Jul 16 21:48:07.079026 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:07.078984 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh"] Jul 16 21:48:07.080940 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:48:07.080902 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7fe99c_9b73_4956_b173_095a8f504b3a.slice/crio-fd4efabde0e8749bcef3418526f70b4f75df171180ca6964538dd5aff999f24f WatchSource:0}: Error finding container fd4efabde0e8749bcef3418526f70b4f75df171180ca6964538dd5aff999f24f: Status 404 returned error can't find the container with id fd4efabde0e8749bcef3418526f70b4f75df171180ca6964538dd5aff999f24f Jul 16 21:48:07.171909 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:07.171881 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh" event={"ID":"3f7fe99c-9b73-4956-b173-095a8f504b3a","Type":"ContainerStarted","Data":"fd4efabde0e8749bcef3418526f70b4f75df171180ca6964538dd5aff999f24f"} Jul 16 21:48:08.177914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:08.177882 2577 generic.go:356] "Generic (PLEG): container finished" podID="3f7fe99c-9b73-4956-b173-095a8f504b3a" containerID="566636f57c666a10a9d2b28440a7f33eff7bdc66fc8e9bb4f7bf671f344c2489" exitCode=6 Jul 16 21:48:08.178326 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:08.177929 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh" event={"ID":"3f7fe99c-9b73-4956-b173-095a8f504b3a","Type":"ContainerDied","Data":"566636f57c666a10a9d2b28440a7f33eff7bdc66fc8e9bb4f7bf671f344c2489"} Jul 16 21:48:08.178326 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:08.178134 2577 scope.go:117] "RemoveContainer" containerID="566636f57c666a10a9d2b28440a7f33eff7bdc66fc8e9bb4f7bf671f344c2489" Jul 16 21:48:09.184774 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:09.184740 2577 generic.go:356] "Generic (PLEG): container finished" podID="3f7fe99c-9b73-4956-b173-095a8f504b3a" containerID="988e626a480eeccb85a082db22a1e051e5eeaf8f19646bc6bff5dbee523dd049" exitCode=6 Jul 16 21:48:09.185169 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:09.184824 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh" event={"ID":"3f7fe99c-9b73-4956-b173-095a8f504b3a","Type":"ContainerDied","Data":"988e626a480eeccb85a082db22a1e051e5eeaf8f19646bc6bff5dbee523dd049"} Jul 16 21:48:09.185169 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:09.184889 2577 scope.go:117] "RemoveContainer" containerID="566636f57c666a10a9d2b28440a7f33eff7bdc66fc8e9bb4f7bf671f344c2489" Jul 16 21:48:09.185169 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:09.185060 2577 scope.go:117] "RemoveContainer" containerID="988e626a480eeccb85a082db22a1e051e5eeaf8f19646bc6bff5dbee523dd049" Jul 16 21:48:09.185352 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:09.185322 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh_odh-ai-gateway-infra(3f7fe99c-9b73-4956-b173-095a8f504b3a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh" podUID="3f7fe99c-9b73-4956-b173-095a8f504b3a" Jul 16 21:48:10.191064 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:10.190987 2577 scope.go:117] "RemoveContainer" containerID="988e626a480eeccb85a082db22a1e051e5eeaf8f19646bc6bff5dbee523dd049" Jul 16 21:48:10.191444 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:10.191200 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh_odh-ai-gateway-infra(3f7fe99c-9b73-4956-b173-095a8f504b3a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh" podUID="3f7fe99c-9b73-4956-b173-095a8f504b3a" Jul 16 21:48:10.971535 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:10.971502 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg"] Jul 16 21:48:11.103693 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:11.103667 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:48:11.129044 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:11.129016 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454a4f80-c2dd-45b4-9b59-9152db8367d1-maas-api-tls\") pod \"454a4f80-c2dd-45b4-9b59-9152db8367d1\" (UID: \"454a4f80-c2dd-45b4-9b59-9152db8367d1\") " Jul 16 21:48:11.129231 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:11.129100 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b759\" (UniqueName: \"kubernetes.io/projected/454a4f80-c2dd-45b4-9b59-9152db8367d1-kube-api-access-4b759\") pod \"454a4f80-c2dd-45b4-9b59-9152db8367d1\" (UID: \"454a4f80-c2dd-45b4-9b59-9152db8367d1\") " Jul 16 21:48:11.129231 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:11.129179 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454a4f80-c2dd-45b4-9b59-9152db8367d1-openshift-service-ca\") pod \"454a4f80-c2dd-45b4-9b59-9152db8367d1\" (UID: \"454a4f80-c2dd-45b4-9b59-9152db8367d1\") " Jul 16 21:48:11.129479 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:11.129457 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454a4f80-c2dd-45b4-9b59-9152db8367d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "454a4f80-c2dd-45b4-9b59-9152db8367d1" (UID: "454a4f80-c2dd-45b4-9b59-9152db8367d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:48:11.130989 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:11.130967 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/454a4f80-c2dd-45b4-9b59-9152db8367d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "454a4f80-c2dd-45b4-9b59-9152db8367d1" (UID: "454a4f80-c2dd-45b4-9b59-9152db8367d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:48:11.131301 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:11.131278 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454a4f80-c2dd-45b4-9b59-9152db8367d1-kube-api-access-4b759" (OuterVolumeSpecName: "kube-api-access-4b759") pod "454a4f80-c2dd-45b4-9b59-9152db8367d1" (UID: "454a4f80-c2dd-45b4-9b59-9152db8367d1"). InnerVolumeSpecName "kube-api-access-4b759". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:48:11.196792 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:11.196771 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" Jul 16 21:48:11.196792 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:11.196782 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg" event={"ID":"454a4f80-c2dd-45b4-9b59-9152db8367d1","Type":"ContainerDied","Data":"5976bf70d5520380a56a9d1597cd8c50070e057cc7c000f50b32fe58f767c70f"} Jul 16 21:48:11.197258 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:11.196839 2577 scope.go:117] "RemoveContainer" containerID="085992d52f8f3be350529602b7ff0fcb1bb0ec6236b05dbde6ccbf46463b3029" Jul 16 21:48:11.218784 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:11.218759 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg"] Jul 16 21:48:11.222475 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:11.222426 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f60157c3-54749c956b-dswbg"] Jul 16 21:48:11.230513 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:11.230496 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454a4f80-c2dd-45b4-9b59-9152db8367d1-openshift-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:48:11.230591 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:11.230515 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454a4f80-c2dd-45b4-9b59-9152db8367d1-maas-api-tls\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:48:11.230591 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:11.230526 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b759\" (UniqueName: \"kubernetes.io/projected/454a4f80-c2dd-45b4-9b59-9152db8367d1-kube-api-access-4b759\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:48:12.449957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:12.449923 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" path="/var/lib/kubelet/pods/454a4f80-c2dd-45b4-9b59-9152db8367d1/volumes" Jul 16 21:48:13.445687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:13.445657 2577 scope.go:117] "RemoveContainer" containerID="cf1a3dd5ac3ba33f8372370886f9cfd2a35e18c43fbfcc164fc126b1016a1c99" Jul 16 21:48:14.214544 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:14.214513 2577 generic.go:356] "Generic (PLEG): container finished" podID="2ca3a37e-1f57-485a-9478-6f6539161f48" containerID="a0526962f6138deb9b59875773c61c94db2f8b61ac8b761ac54d28eca2b03cb9" exitCode=6 Jul 16 21:48:14.214961 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:14.214555 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb" event={"ID":"2ca3a37e-1f57-485a-9478-6f6539161f48","Type":"ContainerDied","Data":"a0526962f6138deb9b59875773c61c94db2f8b61ac8b761ac54d28eca2b03cb9"} Jul 16 21:48:14.214961 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:14.214609 2577 scope.go:117] "RemoveContainer" containerID="cf1a3dd5ac3ba33f8372370886f9cfd2a35e18c43fbfcc164fc126b1016a1c99" Jul 16 21:48:14.214961 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:14.214858 2577 scope.go:117] "RemoveContainer" containerID="a0526962f6138deb9b59875773c61c94db2f8b61ac8b761ac54d28eca2b03cb9" Jul 16 21:48:14.215122 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:14.215100 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb_odh-ai-gateway-infra(2ca3a37e-1f57-485a-9478-6f6539161f48)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb" podUID="2ca3a37e-1f57-485a-9478-6f6539161f48" Jul 16 21:48:15.243067 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:15.243033 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb"] Jul 16 21:48:15.370749 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:15.370726 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb" Jul 16 21:48:15.469967 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:15.469935 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca3a37e-1f57-485a-9478-6f6539161f48-maas-api-service-ca\") pod \"2ca3a37e-1f57-485a-9478-6f6539161f48\" (UID: \"2ca3a37e-1f57-485a-9478-6f6539161f48\") " Jul 16 21:48:15.470326 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:15.470302 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca3a37e-1f57-485a-9478-6f6539161f48-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ca3a37e-1f57-485a-9478-6f6539161f48" (UID: "2ca3a37e-1f57-485a-9478-6f6539161f48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:48:15.571483 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:15.571397 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca3a37e-1f57-485a-9478-6f6539161f48-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:48:16.227046 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:16.227009 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb" event={"ID":"2ca3a37e-1f57-485a-9478-6f6539161f48","Type":"ContainerDied","Data":"cc6c3400b892f45c4798582652dd48c4d97084dc786b65b13a551085d0c27657"} Jul 16 21:48:16.227046 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:16.227031 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb" Jul 16 21:48:16.227300 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:16.227074 2577 scope.go:117] "RemoveContainer" containerID="a0526962f6138deb9b59875773c61c94db2f8b61ac8b761ac54d28eca2b03cb9" Jul 16 21:48:16.250008 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:16.249986 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb"] Jul 16 21:48:16.255653 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:16.255629 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-zd9vb"] Jul 16 21:48:16.450655 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:16.450627 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ca3a37e-1f57-485a-9478-6f6539161f48" path="/var/lib/kubelet/pods/2ca3a37e-1f57-485a-9478-6f6539161f48/volumes" Jul 16 21:48:18.823491 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.823448 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp"] Jul 16 21:48:18.823971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.823948 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:18.823971 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.823967 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:18.824100 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.823987 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:18.824100 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.823996 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:18.824100 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824007 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca3a37e-1f57-485a-9478-6f6539161f48" containerName="revoke-keys" Jul 16 21:48:18.824100 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824017 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca3a37e-1f57-485a-9478-6f6539161f48" containerName="revoke-keys" Jul 16 21:48:18.824100 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824034 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca3a37e-1f57-485a-9478-6f6539161f48" containerName="revoke-keys" Jul 16 21:48:18.824100 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824043 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca3a37e-1f57-485a-9478-6f6539161f48" containerName="revoke-keys" Jul 16 21:48:18.824100 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824054 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca3a37e-1f57-485a-9478-6f6539161f48" containerName="revoke-keys" Jul 16 21:48:18.824100 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824063 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca3a37e-1f57-485a-9478-6f6539161f48" containerName="revoke-keys" Jul 16 21:48:18.824100 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824073 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:18.824100 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824081 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:18.824100 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824092 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:18.824100 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824100 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:18.824695 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824111 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:18.824695 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824119 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:18.824695 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824231 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:18.824695 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824243 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:18.824695 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824251 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca3a37e-1f57-485a-9478-6f6539161f48" containerName="revoke-keys" Jul 16 21:48:18.824695 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824261 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca3a37e-1f57-485a-9478-6f6539161f48" containerName="revoke-keys" Jul 16 21:48:18.824695 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824269 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca3a37e-1f57-485a-9478-6f6539161f48" containerName="revoke-keys" Jul 16 21:48:18.824695 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824277 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:18.824695 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824286 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:18.824695 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.824296 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:18.828825 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.828805 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:48:18.832421 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.832395 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xcg5c\"" Jul 16 21:48:18.832507 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.832423 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c1cf683e-serving-cert\"" Jul 16 21:48:18.833281 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.833262 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 21:48:18.843008 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.842984 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp"] Jul 16 21:48:18.902389 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.902362 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntrff\" (UniqueName: \"kubernetes.io/projected/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-kube-api-access-ntrff\") pod \"maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp\" (UID: \"9fa7af33-c34c-479b-ad5b-d451bdf6b61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:48:18.902547 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.902425 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-openshift-service-ca\") pod \"maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp\" (UID: \"9fa7af33-c34c-479b-ad5b-d451bdf6b61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:48:18.902547 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:18.902454 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-maas-api-tls\") pod \"maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp\" (UID: \"9fa7af33-c34c-479b-ad5b-d451bdf6b61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:48:19.003239 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:19.003205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntrff\" (UniqueName: \"kubernetes.io/projected/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-kube-api-access-ntrff\") pod \"maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp\" (UID: \"9fa7af33-c34c-479b-ad5b-d451bdf6b61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:48:19.003430 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:19.003274 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-openshift-service-ca\") pod \"maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp\" (UID: \"9fa7af33-c34c-479b-ad5b-d451bdf6b61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:48:19.003430 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:19.003303 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-maas-api-tls\") pod \"maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp\" (UID: \"9fa7af33-c34c-479b-ad5b-d451bdf6b61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:48:19.003525 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:19.003491 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-serving-cert: secret "maas-api-e2e-derive-c1cf683e-serving-cert" not found Jul 16 21:48:19.003568 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:19.003558 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-maas-api-tls podName:9fa7af33-c34c-479b-ad5b-d451bdf6b61b nodeName:}" failed. No retries permitted until 2026-07-16 21:48:19.503541827 +0000 UTC m=+2315.678650007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-maas-api-tls") pod "maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" (UID: "9fa7af33-c34c-479b-ad5b-d451bdf6b61b") : secret "maas-api-e2e-derive-c1cf683e-serving-cert" not found Jul 16 21:48:19.003858 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:19.003839 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-openshift-service-ca\") pod \"maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp\" (UID: \"9fa7af33-c34c-479b-ad5b-d451bdf6b61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:48:19.022283 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:19.022263 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntrff\" (UniqueName: \"kubernetes.io/projected/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-kube-api-access-ntrff\") pod \"maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp\" (UID: \"9fa7af33-c34c-479b-ad5b-d451bdf6b61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:48:19.507920 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:19.507884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-maas-api-tls\") pod \"maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp\" (UID: \"9fa7af33-c34c-479b-ad5b-d451bdf6b61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:48:19.510311 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:19.510283 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-maas-api-tls\") pod \"maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp\" (UID: \"9fa7af33-c34c-479b-ad5b-d451bdf6b61b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:48:19.738340 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:19.738304 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:48:19.886677 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:19.886652 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp"] Jul 16 21:48:19.888749 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:48:19.888720 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa7af33_c34c_479b_ad5b_d451bdf6b61b.slice/crio-031a42aba7a24998661fae38857df37b0c254e23fd6c3a39d91c30adb84aa9ae WatchSource:0}: Error finding container 031a42aba7a24998661fae38857df37b0c254e23fd6c3a39d91c30adb84aa9ae: Status 404 returned error can't find the container with id 031a42aba7a24998661fae38857df37b0c254e23fd6c3a39d91c30adb84aa9ae Jul 16 21:48:19.890136 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:19.890115 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:48:20.250735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:20.250705 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" event={"ID":"9fa7af33-c34c-479b-ad5b-d451bdf6b61b","Type":"ContainerStarted","Data":"031a42aba7a24998661fae38857df37b0c254e23fd6c3a39d91c30adb84aa9ae"} Jul 16 21:48:21.259696 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:21.259659 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerID="dff7bc1f45e91742d75f0760059e720b61ce833de7210affae2ef2d1a193baa5" exitCode=1 Jul 16 21:48:21.260142 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:21.259747 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" event={"ID":"9fa7af33-c34c-479b-ad5b-d451bdf6b61b","Type":"ContainerDied","Data":"dff7bc1f45e91742d75f0760059e720b61ce833de7210affae2ef2d1a193baa5"} Jul 16 21:48:21.260142 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:21.259935 2577 scope.go:117] "RemoveContainer" containerID="dff7bc1f45e91742d75f0760059e720b61ce833de7210affae2ef2d1a193baa5" Jul 16 21:48:22.269441 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:22.269404 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerID="7e5ec76184264e5c5220dd60d9d1b364b24fe91bda190b1b65011bd3782853eb" exitCode=1 Jul 16 21:48:22.269911 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:22.269488 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" event={"ID":"9fa7af33-c34c-479b-ad5b-d451bdf6b61b","Type":"ContainerDied","Data":"7e5ec76184264e5c5220dd60d9d1b364b24fe91bda190b1b65011bd3782853eb"} Jul 16 21:48:22.269911 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:22.269558 2577 scope.go:117] "RemoveContainer" containerID="dff7bc1f45e91742d75f0760059e720b61ce833de7210affae2ef2d1a193baa5" Jul 16 21:48:22.269911 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:22.269828 2577 scope.go:117] "RemoveContainer" containerID="7e5ec76184264e5c5220dd60d9d1b364b24fe91bda190b1b65011bd3782853eb" Jul 16 21:48:22.270073 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:22.270050 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:48:22.445590 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:22.445558 2577 scope.go:117] "RemoveContainer" containerID="988e626a480eeccb85a082db22a1e051e5eeaf8f19646bc6bff5dbee523dd049" Jul 16 21:48:23.128280 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.128244 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d"] Jul 16 21:48:23.128610 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.128597 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:23.128654 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.128612 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:23.128654 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.128628 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:23.128654 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.128636 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:23.128754 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.128708 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:23.131750 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.131728 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d" Jul 16 21:48:23.140708 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.140684 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d"] Jul 16 21:48:23.242012 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.241918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f95ee9-adb0-4c53-a0df-ea5e832a3f43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d\" (UID: \"c8f95ee9-adb0-4c53-a0df-ea5e832a3f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d" Jul 16 21:48:23.276590 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.276565 2577 scope.go:117] "RemoveContainer" containerID="7e5ec76184264e5c5220dd60d9d1b364b24fe91bda190b1b65011bd3782853eb" Jul 16 21:48:23.277058 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:23.276770 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:48:23.278054 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.278031 2577 generic.go:356] "Generic (PLEG): container finished" podID="3f7fe99c-9b73-4956-b173-095a8f504b3a" containerID="f8e86694d583600a41a323ffd6b80be1e19284c32cfbe394abff00f15b7ffa7e" exitCode=6 Jul 16 21:48:23.278150 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.278074 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh" event={"ID":"3f7fe99c-9b73-4956-b173-095a8f504b3a","Type":"ContainerDied","Data":"f8e86694d583600a41a323ffd6b80be1e19284c32cfbe394abff00f15b7ffa7e"} Jul 16 21:48:23.278150 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.278119 2577 scope.go:117] "RemoveContainer" containerID="988e626a480eeccb85a082db22a1e051e5eeaf8f19646bc6bff5dbee523dd049" Jul 16 21:48:23.278420 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.278403 2577 scope.go:117] "RemoveContainer" containerID="f8e86694d583600a41a323ffd6b80be1e19284c32cfbe394abff00f15b7ffa7e" Jul 16 21:48:23.278598 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:23.278579 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh_odh-ai-gateway-infra(3f7fe99c-9b73-4956-b173-095a8f504b3a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh" podUID="3f7fe99c-9b73-4956-b173-095a8f504b3a" Jul 16 21:48:23.342659 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.342619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f95ee9-adb0-4c53-a0df-ea5e832a3f43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d\" (UID: \"c8f95ee9-adb0-4c53-a0df-ea5e832a3f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d" Jul 16 21:48:23.343331 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.343307 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f95ee9-adb0-4c53-a0df-ea5e832a3f43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d\" (UID: \"c8f95ee9-adb0-4c53-a0df-ea5e832a3f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d" Jul 16 21:48:23.443442 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.443410 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d" Jul 16 21:48:23.576265 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:23.576239 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d"] Jul 16 21:48:23.578050 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:48:23.578024 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f95ee9_adb0_4c53_a0df_ea5e832a3f43.slice/crio-ccaa570db3b3b141e322b2a9896d618b0e682107e5c4e017a1be74a70a9515e0 WatchSource:0}: Error finding container ccaa570db3b3b141e322b2a9896d618b0e682107e5c4e017a1be74a70a9515e0: Status 404 returned error can't find the container with id ccaa570db3b3b141e322b2a9896d618b0e682107e5c4e017a1be74a70a9515e0 Jul 16 21:48:24.286203 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.286153 2577 generic.go:356] "Generic (PLEG): container finished" podID="c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" containerID="3aa643f63b478e7a83dd100d0c93f503fa1a484f8589f854ca78a5cec2bb9a12" exitCode=7 Jul 16 21:48:24.286584 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.286229 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d" event={"ID":"c8f95ee9-adb0-4c53-a0df-ea5e832a3f43","Type":"ContainerDied","Data":"3aa643f63b478e7a83dd100d0c93f503fa1a484f8589f854ca78a5cec2bb9a12"} Jul 16 21:48:24.286584 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.286271 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d" event={"ID":"c8f95ee9-adb0-4c53-a0df-ea5e832a3f43","Type":"ContainerStarted","Data":"ccaa570db3b3b141e322b2a9896d618b0e682107e5c4e017a1be74a70a9515e0"} Jul 16 21:48:24.286584 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.286456 2577 scope.go:117] "RemoveContainer" containerID="3aa643f63b478e7a83dd100d0c93f503fa1a484f8589f854ca78a5cec2bb9a12" Jul 16 21:48:24.332963 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.332930 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh"] Jul 16 21:48:24.458083 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.458054 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh" Jul 16 21:48:24.554473 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.554430 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7fe99c-9b73-4956-b173-095a8f504b3a-maas-api-service-ca\") pod \"3f7fe99c-9b73-4956-b173-095a8f504b3a\" (UID: \"3f7fe99c-9b73-4956-b173-095a8f504b3a\") " Jul 16 21:48:24.554813 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.554785 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7fe99c-9b73-4956-b173-095a8f504b3a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f7fe99c-9b73-4956-b173-095a8f504b3a" (UID: "3f7fe99c-9b73-4956-b173-095a8f504b3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:48:24.655199 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.655151 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7fe99c-9b73-4956-b173-095a8f504b3a-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:48:24.738888 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.738809 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:48:24.739179 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.739165 2577 scope.go:117] "RemoveContainer" containerID="7e5ec76184264e5c5220dd60d9d1b364b24fe91bda190b1b65011bd3782853eb" Jul 16 21:48:24.739388 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:24.739370 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:48:24.872254 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.872215 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q"] Jul 16 21:48:24.872589 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.872575 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f7fe99c-9b73-4956-b173-095a8f504b3a" containerName="revoke-keys" Jul 16 21:48:24.872673 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.872592 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f7fe99c-9b73-4956-b173-095a8f504b3a" containerName="revoke-keys" Jul 16 21:48:24.872673 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.872611 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f7fe99c-9b73-4956-b173-095a8f504b3a" containerName="revoke-keys" Jul 16 21:48:24.872673 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.872616 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f7fe99c-9b73-4956-b173-095a8f504b3a" containerName="revoke-keys" Jul 16 21:48:24.872673 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.872625 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f7fe99c-9b73-4956-b173-095a8f504b3a" containerName="revoke-keys" Jul 16 21:48:24.872673 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.872631 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f7fe99c-9b73-4956-b173-095a8f504b3a" containerName="revoke-keys" Jul 16 21:48:24.872835 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.872689 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="454a4f80-c2dd-45b4-9b59-9152db8367d1" containerName="maas-api" Jul 16 21:48:24.872835 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.872698 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f7fe99c-9b73-4956-b173-095a8f504b3a" containerName="revoke-keys" Jul 16 21:48:24.872835 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.872705 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f7fe99c-9b73-4956-b173-095a8f504b3a" containerName="revoke-keys" Jul 16 21:48:24.875637 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.875622 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q" Jul 16 21:48:24.884631 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.884601 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q"] Jul 16 21:48:24.957461 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:24.957414 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbf6ec6-684e-4116-87e6-d753d9b05306-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-9j85q\" (UID: \"ccbf6ec6-684e-4116-87e6-d753d9b05306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q" Jul 16 21:48:25.058886 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:25.058853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbf6ec6-684e-4116-87e6-d753d9b05306-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-9j85q\" (UID: \"ccbf6ec6-684e-4116-87e6-d753d9b05306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q" Jul 16 21:48:25.059470 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:25.059452 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbf6ec6-684e-4116-87e6-d753d9b05306-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-9j85q\" (UID: \"ccbf6ec6-684e-4116-87e6-d753d9b05306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q" Jul 16 21:48:25.187232 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:25.187177 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q" Jul 16 21:48:25.295831 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:25.295797 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh" Jul 16 21:48:25.295831 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:25.295810 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh" event={"ID":"3f7fe99c-9b73-4956-b173-095a8f504b3a","Type":"ContainerDied","Data":"fd4efabde0e8749bcef3418526f70b4f75df171180ca6964538dd5aff999f24f"} Jul 16 21:48:25.296313 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:25.295879 2577 scope.go:117] "RemoveContainer" containerID="f8e86694d583600a41a323ffd6b80be1e19284c32cfbe394abff00f15b7ffa7e" Jul 16 21:48:25.298225 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:25.298169 2577 generic.go:356] "Generic (PLEG): container finished" podID="c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" containerID="72734f4cbd92f593542423106411adc6860ec600ff56221017f8266b748a19fd" exitCode=7 Jul 16 21:48:25.298349 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:25.298289 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d" event={"ID":"c8f95ee9-adb0-4c53-a0df-ea5e832a3f43","Type":"ContainerDied","Data":"72734f4cbd92f593542423106411adc6860ec600ff56221017f8266b748a19fd"} Jul 16 21:48:25.298619 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:25.298601 2577 scope.go:117] "RemoveContainer" containerID="72734f4cbd92f593542423106411adc6860ec600ff56221017f8266b748a19fd" Jul 16 21:48:25.298831 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:25.298812 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d_odh-ai-gateway-infra(c8f95ee9-adb0-4c53-a0df-ea5e832a3f43)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d" podUID="c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" Jul 16 21:48:25.306289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:25.306272 2577 scope.go:117] "RemoveContainer" containerID="3aa643f63b478e7a83dd100d0c93f503fa1a484f8589f854ca78a5cec2bb9a12" Jul 16 21:48:25.322475 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:25.322454 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q"] Jul 16 21:48:25.324304 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:48:25.324283 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbf6ec6_684e_4116_87e6_d753d9b05306.slice/crio-8eaae3d0c6f399ab3412a9bef722a887f0fce517d9a77cf404df6a8eabfc160b WatchSource:0}: Error finding container 8eaae3d0c6f399ab3412a9bef722a887f0fce517d9a77cf404df6a8eabfc160b: Status 404 returned error can't find the container with id 8eaae3d0c6f399ab3412a9bef722a887f0fce517d9a77cf404df6a8eabfc160b Jul 16 21:48:25.337231 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:25.337202 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh"] Jul 16 21:48:25.348554 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:25.348529 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-czqmh"] Jul 16 21:48:26.305434 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:26.305352 2577 scope.go:117] "RemoveContainer" containerID="72734f4cbd92f593542423106411adc6860ec600ff56221017f8266b748a19fd" Jul 16 21:48:26.305842 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:26.305568 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d_odh-ai-gateway-infra(c8f95ee9-adb0-4c53-a0df-ea5e832a3f43)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d" podUID="c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" Jul 16 21:48:26.306602 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:26.306576 2577 generic.go:356] "Generic (PLEG): container finished" podID="ccbf6ec6-684e-4116-87e6-d753d9b05306" containerID="4dbcd36beb2d0c1954fd9689ec6063947f2fb36ce50262ea06e83609c5ec719d" exitCode=6 Jul 16 21:48:26.306701 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:26.306661 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q" event={"ID":"ccbf6ec6-684e-4116-87e6-d753d9b05306","Type":"ContainerDied","Data":"4dbcd36beb2d0c1954fd9689ec6063947f2fb36ce50262ea06e83609c5ec719d"} Jul 16 21:48:26.306759 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:26.306715 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q" event={"ID":"ccbf6ec6-684e-4116-87e6-d753d9b05306","Type":"ContainerStarted","Data":"8eaae3d0c6f399ab3412a9bef722a887f0fce517d9a77cf404df6a8eabfc160b"} Jul 16 21:48:26.306808 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:26.306795 2577 scope.go:117] "RemoveContainer" containerID="4dbcd36beb2d0c1954fd9689ec6063947f2fb36ce50262ea06e83609c5ec719d" Jul 16 21:48:26.450716 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:26.450685 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f7fe99c-9b73-4956-b173-095a8f504b3a" path="/var/lib/kubelet/pods/3f7fe99c-9b73-4956-b173-095a8f504b3a/volumes" Jul 16 21:48:27.313168 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:27.313082 2577 generic.go:356] "Generic (PLEG): container finished" podID="ccbf6ec6-684e-4116-87e6-d753d9b05306" containerID="17ee1b427eedb4884a53be0d777dc86bcdae25c18c7ceefdc5f4a310cd79895a" exitCode=6 Jul 16 21:48:27.313578 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:27.313163 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q" event={"ID":"ccbf6ec6-684e-4116-87e6-d753d9b05306","Type":"ContainerDied","Data":"17ee1b427eedb4884a53be0d777dc86bcdae25c18c7ceefdc5f4a310cd79895a"} Jul 16 21:48:27.313578 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:27.313238 2577 scope.go:117] "RemoveContainer" containerID="4dbcd36beb2d0c1954fd9689ec6063947f2fb36ce50262ea06e83609c5ec719d" Jul 16 21:48:27.313578 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:27.313404 2577 scope.go:117] "RemoveContainer" containerID="17ee1b427eedb4884a53be0d777dc86bcdae25c18c7ceefdc5f4a310cd79895a" Jul 16 21:48:27.313706 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:27.313647 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-9j85q_odh-ai-gateway-infra(ccbf6ec6-684e-4116-87e6-d753d9b05306)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q" podUID="ccbf6ec6-684e-4116-87e6-d753d9b05306" Jul 16 21:48:28.327005 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:28.326971 2577 scope.go:117] "RemoveContainer" containerID="17ee1b427eedb4884a53be0d777dc86bcdae25c18c7ceefdc5f4a310cd79895a" Jul 16 21:48:28.327475 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:28.327418 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-9j85q_odh-ai-gateway-infra(ccbf6ec6-684e-4116-87e6-d753d9b05306)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q" podUID="ccbf6ec6-684e-4116-87e6-d753d9b05306" Jul 16 21:48:29.738909 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:29.738860 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:48:29.739374 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:29.739357 2577 scope.go:117] "RemoveContainer" containerID="7e5ec76184264e5c5220dd60d9d1b364b24fe91bda190b1b65011bd3782853eb" Jul 16 21:48:29.739549 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:29.739531 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:48:40.444888 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:40.444809 2577 scope.go:117] "RemoveContainer" containerID="72734f4cbd92f593542423106411adc6860ec600ff56221017f8266b748a19fd" Jul 16 21:48:41.396988 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:41.396957 2577 generic.go:356] "Generic (PLEG): container finished" podID="c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" containerID="d8875b399a24b3bed675c495bd7ff8a1a5eed75f784c816d9d0cc220c727b6fc" exitCode=7 Jul 16 21:48:41.397158 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:41.397031 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d" event={"ID":"c8f95ee9-adb0-4c53-a0df-ea5e832a3f43","Type":"ContainerDied","Data":"d8875b399a24b3bed675c495bd7ff8a1a5eed75f784c816d9d0cc220c727b6fc"} Jul 16 21:48:41.397158 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:41.397088 2577 scope.go:117] "RemoveContainer" containerID="72734f4cbd92f593542423106411adc6860ec600ff56221017f8266b748a19fd" Jul 16 21:48:41.397435 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:41.397417 2577 scope.go:117] "RemoveContainer" containerID="d8875b399a24b3bed675c495bd7ff8a1a5eed75f784c816d9d0cc220c727b6fc" Jul 16 21:48:41.397632 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:41.397613 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d_odh-ai-gateway-infra(c8f95ee9-adb0-4c53-a0df-ea5e832a3f43)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d" podUID="c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" Jul 16 21:48:41.445717 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:41.445687 2577 scope.go:117] "RemoveContainer" containerID="17ee1b427eedb4884a53be0d777dc86bcdae25c18c7ceefdc5f4a310cd79895a" Jul 16 21:48:41.446088 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:41.445807 2577 scope.go:117] "RemoveContainer" containerID="7e5ec76184264e5c5220dd60d9d1b364b24fe91bda190b1b65011bd3782853eb" Jul 16 21:48:42.406157 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:42.406123 2577 generic.go:356] "Generic (PLEG): container finished" podID="ccbf6ec6-684e-4116-87e6-d753d9b05306" containerID="090b093b16310196ffc276307c047f62372e22f5fa6fde1556ff331bfefa03a3" exitCode=6 Jul 16 21:48:42.406368 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:42.406219 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q" event={"ID":"ccbf6ec6-684e-4116-87e6-d753d9b05306","Type":"ContainerDied","Data":"090b093b16310196ffc276307c047f62372e22f5fa6fde1556ff331bfefa03a3"} Jul 16 21:48:42.406368 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:42.406285 2577 scope.go:117] "RemoveContainer" containerID="17ee1b427eedb4884a53be0d777dc86bcdae25c18c7ceefdc5f4a310cd79895a" Jul 16 21:48:42.406587 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:42.406568 2577 scope.go:117] "RemoveContainer" containerID="090b093b16310196ffc276307c047f62372e22f5fa6fde1556ff331bfefa03a3" Jul 16 21:48:42.406840 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:42.406813 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-9j85q_odh-ai-gateway-infra(ccbf6ec6-684e-4116-87e6-d753d9b05306)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q" podUID="ccbf6ec6-684e-4116-87e6-d753d9b05306" Jul 16 21:48:42.408086 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:42.408065 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerID="04493585224a055c3bd87e4c92699bb3f315cab2880f50ba43ee678e5609ef78" exitCode=1 Jul 16 21:48:42.408173 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:42.408143 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" event={"ID":"9fa7af33-c34c-479b-ad5b-d451bdf6b61b","Type":"ContainerDied","Data":"04493585224a055c3bd87e4c92699bb3f315cab2880f50ba43ee678e5609ef78"} Jul 16 21:48:42.408475 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:42.408459 2577 scope.go:117] "RemoveContainer" containerID="04493585224a055c3bd87e4c92699bb3f315cab2880f50ba43ee678e5609ef78" Jul 16 21:48:42.408660 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:42.408642 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:48:42.417530 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:42.417513 2577 scope.go:117] "RemoveContainer" containerID="7e5ec76184264e5c5220dd60d9d1b364b24fe91bda190b1b65011bd3782853eb" Jul 16 21:48:42.425448 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:42.425079 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d"] Jul 16 21:48:42.560145 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:42.560118 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d" Jul 16 21:48:42.626894 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:42.626862 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f95ee9-adb0-4c53-a0df-ea5e832a3f43-maas-api-service-ca\") pod \"c8f95ee9-adb0-4c53-a0df-ea5e832a3f43\" (UID: \"c8f95ee9-adb0-4c53-a0df-ea5e832a3f43\") " Jul 16 21:48:42.627235 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:42.627213 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f95ee9-adb0-4c53-a0df-ea5e832a3f43-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" (UID: "c8f95ee9-adb0-4c53-a0df-ea5e832a3f43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:48:42.728509 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:42.728417 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f95ee9-adb0-4c53-a0df-ea5e832a3f43-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:48:43.415063 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:43.415031 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d" Jul 16 21:48:43.415282 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:43.415030 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d" event={"ID":"c8f95ee9-adb0-4c53-a0df-ea5e832a3f43","Type":"ContainerDied","Data":"ccaa570db3b3b141e322b2a9896d618b0e682107e5c4e017a1be74a70a9515e0"} Jul 16 21:48:43.415282 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:43.415201 2577 scope.go:117] "RemoveContainer" containerID="d8875b399a24b3bed675c495bd7ff8a1a5eed75f784c816d9d0cc220c727b6fc" Jul 16 21:48:43.436662 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:43.436631 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q"] Jul 16 21:48:43.439616 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:43.439596 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d"] Jul 16 21:48:43.447743 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:43.447715 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-bdk4d"] Jul 16 21:48:43.573937 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:43.573913 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q" Jul 16 21:48:43.636410 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:43.636386 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbf6ec6-684e-4116-87e6-d753d9b05306-maas-api-service-ca\") pod \"ccbf6ec6-684e-4116-87e6-d753d9b05306\" (UID: \"ccbf6ec6-684e-4116-87e6-d753d9b05306\") " Jul 16 21:48:43.636723 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:43.636699 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbf6ec6-684e-4116-87e6-d753d9b05306-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ccbf6ec6-684e-4116-87e6-d753d9b05306" (UID: "ccbf6ec6-684e-4116-87e6-d753d9b05306"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:48:43.737033 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:43.736942 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbf6ec6-684e-4116-87e6-d753d9b05306-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:48:44.425288 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:44.425259 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q" Jul 16 21:48:44.425467 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:44.425300 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q" event={"ID":"ccbf6ec6-684e-4116-87e6-d753d9b05306","Type":"ContainerDied","Data":"8eaae3d0c6f399ab3412a9bef722a887f0fce517d9a77cf404df6a8eabfc160b"} Jul 16 21:48:44.425467 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:44.425386 2577 scope.go:117] "RemoveContainer" containerID="090b093b16310196ffc276307c047f62372e22f5fa6fde1556ff331bfefa03a3" Jul 16 21:48:44.451431 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:44.451406 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" path="/var/lib/kubelet/pods/c8f95ee9-adb0-4c53-a0df-ea5e832a3f43/volumes" Jul 16 21:48:44.451680 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:44.451667 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q"] Jul 16 21:48:44.459854 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:44.459832 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-9j85q"] Jul 16 21:48:44.738603 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:44.738518 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:48:44.739054 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:44.738961 2577 scope.go:117] "RemoveContainer" containerID="04493585224a055c3bd87e4c92699bb3f315cab2880f50ba43ee678e5609ef78" Jul 16 21:48:44.739226 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:44.739203 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:48:46.449840 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:46.449809 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccbf6ec6-684e-4116-87e6-d753d9b05306" path="/var/lib/kubelet/pods/ccbf6ec6-684e-4116-87e6-d753d9b05306/volumes" Jul 16 21:48:49.739055 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:49.739019 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:48:49.739461 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:49.739406 2577 scope.go:117] "RemoveContainer" containerID="04493585224a055c3bd87e4c92699bb3f315cab2880f50ba43ee678e5609ef78" Jul 16 21:48:49.739585 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:49.739566 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:48:55.551608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.551578 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m"] Jul 16 21:48:55.552085 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.552066 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" containerName="revoke-keys" Jul 16 21:48:55.552199 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.552089 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" containerName="revoke-keys" Jul 16 21:48:55.552199 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.552103 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccbf6ec6-684e-4116-87e6-d753d9b05306" containerName="revoke-keys" Jul 16 21:48:55.552199 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.552111 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccbf6ec6-684e-4116-87e6-d753d9b05306" containerName="revoke-keys" Jul 16 21:48:55.552199 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.552126 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccbf6ec6-684e-4116-87e6-d753d9b05306" containerName="revoke-keys" Jul 16 21:48:55.552199 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.552134 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccbf6ec6-684e-4116-87e6-d753d9b05306" containerName="revoke-keys" Jul 16 21:48:55.552199 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.552151 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" containerName="revoke-keys" Jul 16 21:48:55.552199 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.552159 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" containerName="revoke-keys" Jul 16 21:48:55.552199 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.552200 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccbf6ec6-684e-4116-87e6-d753d9b05306" containerName="revoke-keys" Jul 16 21:48:55.552583 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.552209 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccbf6ec6-684e-4116-87e6-d753d9b05306" containerName="revoke-keys" Jul 16 21:48:55.552583 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.552298 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccbf6ec6-684e-4116-87e6-d753d9b05306" containerName="revoke-keys" Jul 16 21:48:55.552583 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.552313 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f7fe99c-9b73-4956-b173-095a8f504b3a" containerName="revoke-keys" Jul 16 21:48:55.552583 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.552322 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" containerName="revoke-keys" Jul 16 21:48:55.552583 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.552333 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" containerName="revoke-keys" Jul 16 21:48:55.552583 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.552343 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccbf6ec6-684e-4116-87e6-d753d9b05306" containerName="revoke-keys" Jul 16 21:48:55.555519 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.555500 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" Jul 16 21:48:55.558469 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.558449 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:48:55.569044 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.569022 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m"] Jul 16 21:48:55.641635 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.641607 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94194ed-41a6-4769-b44a-8b3c21a57935-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m\" (UID: \"b94194ed-41a6-4769-b44a-8b3c21a57935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" Jul 16 21:48:55.742848 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.742803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94194ed-41a6-4769-b44a-8b3c21a57935-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m\" (UID: \"b94194ed-41a6-4769-b44a-8b3c21a57935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" Jul 16 21:48:55.743457 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.743437 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94194ed-41a6-4769-b44a-8b3c21a57935-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m\" (UID: \"b94194ed-41a6-4769-b44a-8b3c21a57935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" Jul 16 21:48:55.865091 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:55.864996 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" Jul 16 21:48:56.002067 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:56.002035 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m"] Jul 16 21:48:56.004111 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:48:56.004078 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94194ed_41a6_4769_b44a_8b3c21a57935.slice/crio-5808eb529fdc1909940a92cdbc2a2633f7229aa0514cb29838b373f30832745f WatchSource:0}: Error finding container 5808eb529fdc1909940a92cdbc2a2633f7229aa0514cb29838b373f30832745f: Status 404 returned error can't find the container with id 5808eb529fdc1909940a92cdbc2a2633f7229aa0514cb29838b373f30832745f Jul 16 21:48:56.489332 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:56.489241 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" event={"ID":"b94194ed-41a6-4769-b44a-8b3c21a57935","Type":"ContainerStarted","Data":"c5c21646569954f79ab314dfe5227acbfc4f170fa65bdbe7f7c800b765829624"} Jul 16 21:48:56.489332 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:56.489302 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" event={"ID":"b94194ed-41a6-4769-b44a-8b3c21a57935","Type":"ContainerStarted","Data":"5808eb529fdc1909940a92cdbc2a2633f7229aa0514cb29838b373f30832745f"} Jul 16 21:48:56.506202 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:56.506134 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" podStartSLOduration=1.506117852 podStartE2EDuration="1.506117852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:56.505391481 +0000 UTC m=+2352.680499673" watchObservedRunningTime="2026-07-16 21:48:56.506117852 +0000 UTC m=+2352.681226061" Jul 16 21:48:56.681491 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:56.681454 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7"] Jul 16 21:48:56.681861 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:56.681816 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" containerName="revoke-keys" Jul 16 21:48:56.681861 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:56.681827 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" containerName="revoke-keys" Jul 16 21:48:56.681940 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:56.681893 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccbf6ec6-684e-4116-87e6-d753d9b05306" containerName="revoke-keys" Jul 16 21:48:56.681940 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:56.681904 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8f95ee9-adb0-4c53-a0df-ea5e832a3f43" containerName="revoke-keys" Jul 16 21:48:56.684923 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:56.684906 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" Jul 16 21:48:56.693845 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:56.693814 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7"] Jul 16 21:48:56.753289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:56.753173 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7\" (UID: \"ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" Jul 16 21:48:56.853931 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:56.853889 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7\" (UID: \"ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" Jul 16 21:48:56.854598 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:56.854578 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7\" (UID: \"ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" Jul 16 21:48:56.996434 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:56.996399 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" Jul 16 21:48:57.128238 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:57.128203 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7"] Jul 16 21:48:57.129568 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:48:57.129541 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5040bf_bf4f_43cd_bb54_4dab7ed70fc6.slice/crio-98b1ca991b2289cdd60e30e2dc77c18965b2ae782d9576f134d5c230751ed031 WatchSource:0}: Error finding container 98b1ca991b2289cdd60e30e2dc77c18965b2ae782d9576f134d5c230751ed031: Status 404 returned error can't find the container with id 98b1ca991b2289cdd60e30e2dc77c18965b2ae782d9576f134d5c230751ed031 Jul 16 21:48:57.496168 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:57.496127 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" event={"ID":"ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6","Type":"ContainerStarted","Data":"9f06d112808b7263e481303d349b4380f1a6643849bf16f08b403d5cd22a87a3"} Jul 16 21:48:57.496351 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:57.496200 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" event={"ID":"ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6","Type":"ContainerStarted","Data":"98b1ca991b2289cdd60e30e2dc77c18965b2ae782d9576f134d5c230751ed031"} Jul 16 21:48:57.497605 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:57.497575 2577 generic.go:356] "Generic (PLEG): container finished" podID="b94194ed-41a6-4769-b44a-8b3c21a57935" containerID="c5c21646569954f79ab314dfe5227acbfc4f170fa65bdbe7f7c800b765829624" exitCode=6 Jul 16 21:48:57.497705 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:57.497627 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" event={"ID":"b94194ed-41a6-4769-b44a-8b3c21a57935","Type":"ContainerDied","Data":"c5c21646569954f79ab314dfe5227acbfc4f170fa65bdbe7f7c800b765829624"} Jul 16 21:48:57.497832 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:57.497819 2577 scope.go:117] "RemoveContainer" containerID="c5c21646569954f79ab314dfe5227acbfc4f170fa65bdbe7f7c800b765829624" Jul 16 21:48:57.518902 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:57.518853 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" podStartSLOduration=1.5188364810000001 podStartE2EDuration="1.518836481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:57.518068215 +0000 UTC m=+2353.693176406" watchObservedRunningTime="2026-07-16 21:48:57.518836481 +0000 UTC m=+2353.693944672" Jul 16 21:48:58.503880 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:58.503838 2577 generic.go:356] "Generic (PLEG): container finished" podID="b94194ed-41a6-4769-b44a-8b3c21a57935" containerID="0f8134d7eaf78f6828ae8307089746cd2c84ecfb72586e10627d83b98bae79bd" exitCode=6 Jul 16 21:48:58.504357 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:58.503924 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" event={"ID":"b94194ed-41a6-4769-b44a-8b3c21a57935","Type":"ContainerDied","Data":"0f8134d7eaf78f6828ae8307089746cd2c84ecfb72586e10627d83b98bae79bd"} Jul 16 21:48:58.504357 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:58.503986 2577 scope.go:117] "RemoveContainer" containerID="c5c21646569954f79ab314dfe5227acbfc4f170fa65bdbe7f7c800b765829624" Jul 16 21:48:58.504357 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:58.504221 2577 scope.go:117] "RemoveContainer" containerID="0f8134d7eaf78f6828ae8307089746cd2c84ecfb72586e10627d83b98bae79bd" Jul 16 21:48:58.504506 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:58.504442 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m_odh-ai-gateway-infra(b94194ed-41a6-4769-b44a-8b3c21a57935)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" podUID="b94194ed-41a6-4769-b44a-8b3c21a57935" Jul 16 21:48:58.505550 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:58.505527 2577 generic.go:356] "Generic (PLEG): container finished" podID="ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" containerID="9f06d112808b7263e481303d349b4380f1a6643849bf16f08b403d5cd22a87a3" exitCode=6 Jul 16 21:48:58.505700 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:58.505562 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" event={"ID":"ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6","Type":"ContainerDied","Data":"9f06d112808b7263e481303d349b4380f1a6643849bf16f08b403d5cd22a87a3"} Jul 16 21:48:58.505818 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:58.505805 2577 scope.go:117] "RemoveContainer" containerID="9f06d112808b7263e481303d349b4380f1a6643849bf16f08b403d5cd22a87a3" Jul 16 21:48:59.512165 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:59.512130 2577 generic.go:356] "Generic (PLEG): container finished" podID="ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" containerID="6db91b3919c99569335a1902887095f61c26098d8dfd2be19e1a1cc798fa5e41" exitCode=6 Jul 16 21:48:59.512640 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:59.512233 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" event={"ID":"ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6","Type":"ContainerDied","Data":"6db91b3919c99569335a1902887095f61c26098d8dfd2be19e1a1cc798fa5e41"} Jul 16 21:48:59.512640 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:59.512305 2577 scope.go:117] "RemoveContainer" containerID="9f06d112808b7263e481303d349b4380f1a6643849bf16f08b403d5cd22a87a3" Jul 16 21:48:59.512640 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:59.512511 2577 scope.go:117] "RemoveContainer" containerID="6db91b3919c99569335a1902887095f61c26098d8dfd2be19e1a1cc798fa5e41" Jul 16 21:48:59.512832 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:59.512751 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7_odh-ai-gateway-infra(ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" podUID="ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" Jul 16 21:48:59.514063 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:48:59.514031 2577 scope.go:117] "RemoveContainer" containerID="0f8134d7eaf78f6828ae8307089746cd2c84ecfb72586e10627d83b98bae79bd" Jul 16 21:48:59.514233 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:48:59.514216 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m_odh-ai-gateway-infra(b94194ed-41a6-4769-b44a-8b3c21a57935)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" podUID="b94194ed-41a6-4769-b44a-8b3c21a57935" Jul 16 21:49:00.520523 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:00.520498 2577 scope.go:117] "RemoveContainer" containerID="6db91b3919c99569335a1902887095f61c26098d8dfd2be19e1a1cc798fa5e41" Jul 16 21:49:00.520890 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:00.520699 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7_odh-ai-gateway-infra(ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" podUID="ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" Jul 16 21:49:05.444750 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:05.444715 2577 scope.go:117] "RemoveContainer" containerID="04493585224a055c3bd87e4c92699bb3f315cab2880f50ba43ee678e5609ef78" Jul 16 21:49:06.554505 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:06.554393 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerID="1c07562366e45320fefeb92430686b72f3068ed045497950bedf5d5b27cb2daf" exitCode=1 Jul 16 21:49:06.554505 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:06.554455 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" event={"ID":"9fa7af33-c34c-479b-ad5b-d451bdf6b61b","Type":"ContainerDied","Data":"1c07562366e45320fefeb92430686b72f3068ed045497950bedf5d5b27cb2daf"} Jul 16 21:49:06.554968 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:06.554520 2577 scope.go:117] "RemoveContainer" containerID="04493585224a055c3bd87e4c92699bb3f315cab2880f50ba43ee678e5609ef78" Jul 16 21:49:06.554968 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:06.554805 2577 scope.go:117] "RemoveContainer" containerID="1c07562366e45320fefeb92430686b72f3068ed045497950bedf5d5b27cb2daf" Jul 16 21:49:06.555089 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:06.555033 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:49:09.739234 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:09.739199 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:49:09.739647 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:09.739254 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:49:09.739647 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:09.739539 2577 scope.go:117] "RemoveContainer" containerID="1c07562366e45320fefeb92430686b72f3068ed045497950bedf5d5b27cb2daf" Jul 16 21:49:09.739722 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:09.739693 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:49:12.445133 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:12.445093 2577 scope.go:117] "RemoveContainer" containerID="6db91b3919c99569335a1902887095f61c26098d8dfd2be19e1a1cc798fa5e41" Jul 16 21:49:13.444906 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:13.444877 2577 scope.go:117] "RemoveContainer" containerID="0f8134d7eaf78f6828ae8307089746cd2c84ecfb72586e10627d83b98bae79bd" Jul 16 21:49:13.594458 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:13.594431 2577 generic.go:356] "Generic (PLEG): container finished" podID="ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" containerID="2aebd58f88ef1757a45859baeb652cad987f7173601052ddb1a40aa0ce8fcbc9" exitCode=6 Jul 16 21:49:13.594824 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:13.594501 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" event={"ID":"ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6","Type":"ContainerDied","Data":"2aebd58f88ef1757a45859baeb652cad987f7173601052ddb1a40aa0ce8fcbc9"} Jul 16 21:49:13.594824 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:13.594563 2577 scope.go:117] "RemoveContainer" containerID="6db91b3919c99569335a1902887095f61c26098d8dfd2be19e1a1cc798fa5e41" Jul 16 21:49:13.594824 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:13.594811 2577 scope.go:117] "RemoveContainer" containerID="2aebd58f88ef1757a45859baeb652cad987f7173601052ddb1a40aa0ce8fcbc9" Jul 16 21:49:13.595041 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:13.595023 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7_odh-ai-gateway-infra(ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" podUID="ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" Jul 16 21:49:14.600521 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.600492 2577 generic.go:356] "Generic (PLEG): container finished" podID="b94194ed-41a6-4769-b44a-8b3c21a57935" containerID="444cde1459b777c8c560d59ac5b5326ac0c740f74661f2a8b3421383f8ea1fe7" exitCode=6 Jul 16 21:49:14.600919 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.600567 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" event={"ID":"b94194ed-41a6-4769-b44a-8b3c21a57935","Type":"ContainerDied","Data":"444cde1459b777c8c560d59ac5b5326ac0c740f74661f2a8b3421383f8ea1fe7"} Jul 16 21:49:14.600919 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.600632 2577 scope.go:117] "RemoveContainer" containerID="0f8134d7eaf78f6828ae8307089746cd2c84ecfb72586e10627d83b98bae79bd" Jul 16 21:49:14.600997 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.600917 2577 scope.go:117] "RemoveContainer" containerID="444cde1459b777c8c560d59ac5b5326ac0c740f74661f2a8b3421383f8ea1fe7" Jul 16 21:49:14.601163 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:14.601137 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m_odh-ai-gateway-infra(b94194ed-41a6-4769-b44a-8b3c21a57935)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" podUID="b94194ed-41a6-4769-b44a-8b3c21a57935" Jul 16 21:49:14.625507 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.625487 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7"] Jul 16 21:49:14.753870 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.753847 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" Jul 16 21:49:14.913413 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.913384 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6-maas-api-service-ca\") pod \"ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6\" (UID: \"ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6\") " Jul 16 21:49:14.913743 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.913724 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" (UID: "ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:49:14.942282 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.942250 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6"] Jul 16 21:49:14.942657 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.942643 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" containerName="revoke-keys" Jul 16 21:49:14.942718 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.942661 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" containerName="revoke-keys" Jul 16 21:49:14.942718 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.942675 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" containerName="revoke-keys" Jul 16 21:49:14.942718 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.942682 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" containerName="revoke-keys" Jul 16 21:49:14.942828 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.942757 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" containerName="revoke-keys" Jul 16 21:49:14.942828 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.942771 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" containerName="revoke-keys" Jul 16 21:49:14.945736 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.945720 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6" Jul 16 21:49:14.955437 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:14.955411 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6"] Jul 16 21:49:15.014036 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:15.014009 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:49:15.115335 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:15.115306 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c76040f-e10a-4619-9401-90721fd1ad3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6\" (UID: \"8c76040f-e10a-4619-9401-90721fd1ad3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6" Jul 16 21:49:15.216705 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:15.216621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c76040f-e10a-4619-9401-90721fd1ad3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6\" (UID: \"8c76040f-e10a-4619-9401-90721fd1ad3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6" Jul 16 21:49:15.217286 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:15.217269 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c76040f-e10a-4619-9401-90721fd1ad3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6\" (UID: \"8c76040f-e10a-4619-9401-90721fd1ad3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6" Jul 16 21:49:15.256724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:15.256703 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6" Jul 16 21:49:15.385425 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:15.385400 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6"] Jul 16 21:49:15.386949 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:49:15.386921 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c76040f_e10a_4619_9401_90721fd1ad3e.slice/crio-53d5e3b8f65c6ab33d7c85299ecb5221299345f0f3e70d3a24f6d06d3f1d8775 WatchSource:0}: Error finding container 53d5e3b8f65c6ab33d7c85299ecb5221299345f0f3e70d3a24f6d06d3f1d8775: Status 404 returned error can't find the container with id 53d5e3b8f65c6ab33d7c85299ecb5221299345f0f3e70d3a24f6d06d3f1d8775 Jul 16 21:49:15.608507 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:15.608486 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" Jul 16 21:49:15.608903 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:15.608535 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7" event={"ID":"ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6","Type":"ContainerDied","Data":"98b1ca991b2289cdd60e30e2dc77c18965b2ae782d9576f134d5c230751ed031"} Jul 16 21:49:15.608903 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:15.608597 2577 scope.go:117] "RemoveContainer" containerID="2aebd58f88ef1757a45859baeb652cad987f7173601052ddb1a40aa0ce8fcbc9" Jul 16 21:49:15.611606 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:15.611578 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6" event={"ID":"8c76040f-e10a-4619-9401-90721fd1ad3e","Type":"ContainerStarted","Data":"53d5e3b8f65c6ab33d7c85299ecb5221299345f0f3e70d3a24f6d06d3f1d8775"} Jul 16 21:49:15.628712 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:15.628689 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m"] Jul 16 21:49:15.637752 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:15.634831 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7"] Jul 16 21:49:15.641464 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:15.641440 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-5c4q7"] Jul 16 21:49:15.780478 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:15.780456 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" Jul 16 21:49:15.923418 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:15.923388 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94194ed-41a6-4769-b44a-8b3c21a57935-maas-api-service-ca\") pod \"b94194ed-41a6-4769-b44a-8b3c21a57935\" (UID: \"b94194ed-41a6-4769-b44a-8b3c21a57935\") " Jul 16 21:49:15.923737 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:15.923712 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94194ed-41a6-4769-b44a-8b3c21a57935-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b94194ed-41a6-4769-b44a-8b3c21a57935" (UID: "b94194ed-41a6-4769-b44a-8b3c21a57935"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:49:16.024822 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:16.024796 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94194ed-41a6-4769-b44a-8b3c21a57935-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:49:16.450261 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:16.450229 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" path="/var/lib/kubelet/pods/ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6/volumes" Jul 16 21:49:16.619312 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:16.619231 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" Jul 16 21:49:16.619731 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:16.619230 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m" event={"ID":"b94194ed-41a6-4769-b44a-8b3c21a57935","Type":"ContainerDied","Data":"5808eb529fdc1909940a92cdbc2a2633f7229aa0514cb29838b373f30832745f"} Jul 16 21:49:16.619731 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:16.619374 2577 scope.go:117] "RemoveContainer" containerID="444cde1459b777c8c560d59ac5b5326ac0c740f74661f2a8b3421383f8ea1fe7" Jul 16 21:49:16.620873 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:16.620849 2577 generic.go:356] "Generic (PLEG): container finished" podID="8c76040f-e10a-4619-9401-90721fd1ad3e" containerID="080c5472adea8942cf639db7759da953d1ef457b5e9641660a446ea40cdcfec9" exitCode=6 Jul 16 21:49:16.620989 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:16.620925 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6" event={"ID":"8c76040f-e10a-4619-9401-90721fd1ad3e","Type":"ContainerDied","Data":"080c5472adea8942cf639db7759da953d1ef457b5e9641660a446ea40cdcfec9"} Jul 16 21:49:16.621090 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:16.621075 2577 scope.go:117] "RemoveContainer" containerID="080c5472adea8942cf639db7759da953d1ef457b5e9641660a446ea40cdcfec9" Jul 16 21:49:16.656130 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:16.656102 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m"] Jul 16 21:49:16.660149 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:16.660126 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wnf8m"] Jul 16 21:49:17.629043 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:17.629009 2577 generic.go:356] "Generic (PLEG): container finished" podID="8c76040f-e10a-4619-9401-90721fd1ad3e" containerID="93a7922ccd9c6783a30565ab241b143104dc9d4d8a006f3f21fb7f17e05edb53" exitCode=6 Jul 16 21:49:17.629523 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:17.629100 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6" event={"ID":"8c76040f-e10a-4619-9401-90721fd1ad3e","Type":"ContainerDied","Data":"93a7922ccd9c6783a30565ab241b143104dc9d4d8a006f3f21fb7f17e05edb53"} Jul 16 21:49:17.629523 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:17.629207 2577 scope.go:117] "RemoveContainer" containerID="080c5472adea8942cf639db7759da953d1ef457b5e9641660a446ea40cdcfec9" Jul 16 21:49:17.629523 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:17.629303 2577 scope.go:117] "RemoveContainer" containerID="93a7922ccd9c6783a30565ab241b143104dc9d4d8a006f3f21fb7f17e05edb53" Jul 16 21:49:17.629523 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:17.629515 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6_odh-ai-gateway-infra(8c76040f-e10a-4619-9401-90721fd1ad3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6" podUID="8c76040f-e10a-4619-9401-90721fd1ad3e" Jul 16 21:49:18.450393 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:18.450363 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b94194ed-41a6-4769-b44a-8b3c21a57935" path="/var/lib/kubelet/pods/b94194ed-41a6-4769-b44a-8b3c21a57935/volumes" Jul 16 21:49:18.636395 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:18.636371 2577 scope.go:117] "RemoveContainer" containerID="93a7922ccd9c6783a30565ab241b143104dc9d4d8a006f3f21fb7f17e05edb53" Jul 16 21:49:18.636788 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:18.636622 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6_odh-ai-gateway-infra(8c76040f-e10a-4619-9401-90721fd1ad3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6" podUID="8c76040f-e10a-4619-9401-90721fd1ad3e" Jul 16 21:49:21.444772 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:21.444739 2577 scope.go:117] "RemoveContainer" containerID="1c07562366e45320fefeb92430686b72f3068ed045497950bedf5d5b27cb2daf" Jul 16 21:49:21.445179 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:21.444956 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:49:23.177517 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.177479 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29"] Jul 16 21:49:23.178025 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.178005 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b94194ed-41a6-4769-b44a-8b3c21a57935" containerName="revoke-keys" Jul 16 21:49:23.178122 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.178028 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b94194ed-41a6-4769-b44a-8b3c21a57935" containerName="revoke-keys" Jul 16 21:49:23.178122 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.178066 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b94194ed-41a6-4769-b44a-8b3c21a57935" containerName="revoke-keys" Jul 16 21:49:23.178122 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.178074 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b94194ed-41a6-4769-b44a-8b3c21a57935" containerName="revoke-keys" Jul 16 21:49:23.178122 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.178097 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" containerName="revoke-keys" Jul 16 21:49:23.178122 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.178107 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" containerName="revoke-keys" Jul 16 21:49:23.178399 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.178212 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b94194ed-41a6-4769-b44a-8b3c21a57935" containerName="revoke-keys" Jul 16 21:49:23.178399 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.178231 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b94194ed-41a6-4769-b44a-8b3c21a57935" containerName="revoke-keys" Jul 16 21:49:23.178399 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.178241 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab5040bf-bf4f-43cd-bb54-4dab7ed70fc6" containerName="revoke-keys" Jul 16 21:49:23.181252 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.181232 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29" Jul 16 21:49:23.189411 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.188934 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29"] Jul 16 21:49:23.294091 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.294044 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483ade17-adcc-4ebd-ad88-4afd29d8eb8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29\" (UID: \"483ade17-adcc-4ebd-ad88-4afd29d8eb8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29" Jul 16 21:49:23.395219 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.395165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483ade17-adcc-4ebd-ad88-4afd29d8eb8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29\" (UID: \"483ade17-adcc-4ebd-ad88-4afd29d8eb8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29" Jul 16 21:49:23.395784 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.395764 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483ade17-adcc-4ebd-ad88-4afd29d8eb8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29\" (UID: \"483ade17-adcc-4ebd-ad88-4afd29d8eb8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29" Jul 16 21:49:23.492891 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.492817 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29" Jul 16 21:49:23.622377 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.622348 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29"] Jul 16 21:49:23.624370 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:49:23.624346 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483ade17_adcc_4ebd_ad88_4afd29d8eb8d.slice/crio-be96e40a42b1272c8c38f7f6d8ce7174f7c9cabee504e07059f009181d1326e1 WatchSource:0}: Error finding container be96e40a42b1272c8c38f7f6d8ce7174f7c9cabee504e07059f009181d1326e1: Status 404 returned error can't find the container with id be96e40a42b1272c8c38f7f6d8ce7174f7c9cabee504e07059f009181d1326e1 Jul 16 21:49:23.661936 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:23.661908 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29" event={"ID":"483ade17-adcc-4ebd-ad88-4afd29d8eb8d","Type":"ContainerStarted","Data":"be96e40a42b1272c8c38f7f6d8ce7174f7c9cabee504e07059f009181d1326e1"} Jul 16 21:49:24.668564 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:24.668531 2577 generic.go:356] "Generic (PLEG): container finished" podID="483ade17-adcc-4ebd-ad88-4afd29d8eb8d" containerID="865cd72dd5065452830ec930b6332f9183a61db67b3121eb99bdb69fb1e8ef68" exitCode=7 Jul 16 21:49:24.668916 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:24.668589 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29" event={"ID":"483ade17-adcc-4ebd-ad88-4afd29d8eb8d","Type":"ContainerDied","Data":"865cd72dd5065452830ec930b6332f9183a61db67b3121eb99bdb69fb1e8ef68"} Jul 16 21:49:24.668916 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:24.668751 2577 scope.go:117] "RemoveContainer" containerID="865cd72dd5065452830ec930b6332f9183a61db67b3121eb99bdb69fb1e8ef68" Jul 16 21:49:25.675560 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:25.675524 2577 generic.go:356] "Generic (PLEG): container finished" podID="483ade17-adcc-4ebd-ad88-4afd29d8eb8d" containerID="340f3de75db88876a36b582ddfb1953013813aefeee7d08af17c3a1ee3c6bbca" exitCode=7 Jul 16 21:49:25.675937 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:25.675610 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29" event={"ID":"483ade17-adcc-4ebd-ad88-4afd29d8eb8d","Type":"ContainerDied","Data":"340f3de75db88876a36b582ddfb1953013813aefeee7d08af17c3a1ee3c6bbca"} Jul 16 21:49:25.675937 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:25.675658 2577 scope.go:117] "RemoveContainer" containerID="865cd72dd5065452830ec930b6332f9183a61db67b3121eb99bdb69fb1e8ef68" Jul 16 21:49:25.675937 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:25.675892 2577 scope.go:117] "RemoveContainer" containerID="340f3de75db88876a36b582ddfb1953013813aefeee7d08af17c3a1ee3c6bbca" Jul 16 21:49:25.676093 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:25.676074 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29_odh-ai-gateway-infra(483ade17-adcc-4ebd-ad88-4afd29d8eb8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29" podUID="483ade17-adcc-4ebd-ad88-4afd29d8eb8d" Jul 16 21:49:26.681829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:26.681799 2577 scope.go:117] "RemoveContainer" containerID="340f3de75db88876a36b582ddfb1953013813aefeee7d08af17c3a1ee3c6bbca" Jul 16 21:49:26.682253 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:26.682047 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29_odh-ai-gateway-infra(483ade17-adcc-4ebd-ad88-4afd29d8eb8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29" podUID="483ade17-adcc-4ebd-ad88-4afd29d8eb8d" Jul 16 21:49:29.444722 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:29.444691 2577 scope.go:117] "RemoveContainer" containerID="93a7922ccd9c6783a30565ab241b143104dc9d4d8a006f3f21fb7f17e05edb53" Jul 16 21:49:30.702575 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:30.702544 2577 generic.go:356] "Generic (PLEG): container finished" podID="8c76040f-e10a-4619-9401-90721fd1ad3e" containerID="6d451bcac9d36bb35567cf8962f45047850419876e865b4ba2dc82e6bec6f0a6" exitCode=6 Jul 16 21:49:30.703030 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:30.702627 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6" event={"ID":"8c76040f-e10a-4619-9401-90721fd1ad3e","Type":"ContainerDied","Data":"6d451bcac9d36bb35567cf8962f45047850419876e865b4ba2dc82e6bec6f0a6"} Jul 16 21:49:30.703030 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:30.702704 2577 scope.go:117] "RemoveContainer" containerID="93a7922ccd9c6783a30565ab241b143104dc9d4d8a006f3f21fb7f17e05edb53" Jul 16 21:49:30.703160 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:30.703052 2577 scope.go:117] "RemoveContainer" containerID="6d451bcac9d36bb35567cf8962f45047850419876e865b4ba2dc82e6bec6f0a6" Jul 16 21:49:30.703272 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:30.703254 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6_odh-ai-gateway-infra(8c76040f-e10a-4619-9401-90721fd1ad3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6" podUID="8c76040f-e10a-4619-9401-90721fd1ad3e" Jul 16 21:49:31.730609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:31.730579 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6"] Jul 16 21:49:31.864548 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:31.864519 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6" Jul 16 21:49:31.964259 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:31.964223 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c76040f-e10a-4619-9401-90721fd1ad3e-maas-api-service-ca\") pod \"8c76040f-e10a-4619-9401-90721fd1ad3e\" (UID: \"8c76040f-e10a-4619-9401-90721fd1ad3e\") " Jul 16 21:49:31.964562 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:31.964540 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c76040f-e10a-4619-9401-90721fd1ad3e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c76040f-e10a-4619-9401-90721fd1ad3e" (UID: "8c76040f-e10a-4619-9401-90721fd1ad3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:49:32.064905 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:32.064820 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c76040f-e10a-4619-9401-90721fd1ad3e-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:49:32.716827 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:32.716743 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6" event={"ID":"8c76040f-e10a-4619-9401-90721fd1ad3e","Type":"ContainerDied","Data":"53d5e3b8f65c6ab33d7c85299ecb5221299345f0f3e70d3a24f6d06d3f1d8775"} Jul 16 21:49:32.716827 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:32.716778 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6" Jul 16 21:49:32.716827 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:32.716804 2577 scope.go:117] "RemoveContainer" containerID="6d451bcac9d36bb35567cf8962f45047850419876e865b4ba2dc82e6bec6f0a6" Jul 16 21:49:32.737901 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:32.737870 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6"] Jul 16 21:49:32.744080 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:32.744058 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-f4zj6"] Jul 16 21:49:34.451245 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:34.451208 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c76040f-e10a-4619-9401-90721fd1ad3e" path="/var/lib/kubelet/pods/8c76040f-e10a-4619-9401-90721fd1ad3e/volumes" Jul 16 21:49:36.450376 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:36.450352 2577 scope.go:117] "RemoveContainer" containerID="1c07562366e45320fefeb92430686b72f3068ed045497950bedf5d5b27cb2daf" Jul 16 21:49:36.450813 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:36.450548 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:49:38.450906 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:38.450880 2577 scope.go:117] "RemoveContainer" containerID="340f3de75db88876a36b582ddfb1953013813aefeee7d08af17c3a1ee3c6bbca" Jul 16 21:49:39.760325 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:39.760292 2577 generic.go:356] "Generic (PLEG): container finished" podID="483ade17-adcc-4ebd-ad88-4afd29d8eb8d" containerID="f0edda214d95885f9890b98e2def7784211b6935de4228aa9197e70e70fb7e45" exitCode=7 Jul 16 21:49:39.760724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:39.760384 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29" event={"ID":"483ade17-adcc-4ebd-ad88-4afd29d8eb8d","Type":"ContainerDied","Data":"f0edda214d95885f9890b98e2def7784211b6935de4228aa9197e70e70fb7e45"} Jul 16 21:49:39.760724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:39.760463 2577 scope.go:117] "RemoveContainer" containerID="340f3de75db88876a36b582ddfb1953013813aefeee7d08af17c3a1ee3c6bbca" Jul 16 21:49:39.760867 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:39.760763 2577 scope.go:117] "RemoveContainer" containerID="f0edda214d95885f9890b98e2def7784211b6935de4228aa9197e70e70fb7e45" Jul 16 21:49:39.761000 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:39.760974 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29_odh-ai-gateway-infra(483ade17-adcc-4ebd-ad88-4afd29d8eb8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29" podUID="483ade17-adcc-4ebd-ad88-4afd29d8eb8d" Jul 16 21:49:40.789672 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:40.789636 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29"] Jul 16 21:49:40.924162 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:40.924141 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29" Jul 16 21:49:41.043360 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:41.043282 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483ade17-adcc-4ebd-ad88-4afd29d8eb8d-maas-api-service-ca\") pod \"483ade17-adcc-4ebd-ad88-4afd29d8eb8d\" (UID: \"483ade17-adcc-4ebd-ad88-4afd29d8eb8d\") " Jul 16 21:49:41.043641 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:41.043618 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/483ade17-adcc-4ebd-ad88-4afd29d8eb8d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "483ade17-adcc-4ebd-ad88-4afd29d8eb8d" (UID: "483ade17-adcc-4ebd-ad88-4afd29d8eb8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:49:41.144699 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:41.144664 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483ade17-adcc-4ebd-ad88-4afd29d8eb8d-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:49:41.773021 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:41.772987 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29" Jul 16 21:49:41.773214 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:41.773040 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29" event={"ID":"483ade17-adcc-4ebd-ad88-4afd29d8eb8d","Type":"ContainerDied","Data":"be96e40a42b1272c8c38f7f6d8ce7174f7c9cabee504e07059f009181d1326e1"} Jul 16 21:49:41.773214 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:41.773097 2577 scope.go:117] "RemoveContainer" containerID="f0edda214d95885f9890b98e2def7784211b6935de4228aa9197e70e70fb7e45" Jul 16 21:49:41.799750 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:41.799720 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29"] Jul 16 21:49:41.807383 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:41.807358 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-pcn29"] Jul 16 21:49:42.450805 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:42.450775 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="483ade17-adcc-4ebd-ad88-4afd29d8eb8d" path="/var/lib/kubelet/pods/483ade17-adcc-4ebd-ad88-4afd29d8eb8d/volumes" Jul 16 21:49:44.471899 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:44.471869 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:49:44.480247 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:44.480223 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:49:46.731835 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.731792 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn"] Jul 16 21:49:46.732356 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.732299 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="483ade17-adcc-4ebd-ad88-4afd29d8eb8d" containerName="revoke-keys" Jul 16 21:49:46.732356 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.732319 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="483ade17-adcc-4ebd-ad88-4afd29d8eb8d" containerName="revoke-keys" Jul 16 21:49:46.732356 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.732331 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="483ade17-adcc-4ebd-ad88-4afd29d8eb8d" containerName="revoke-keys" Jul 16 21:49:46.732356 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.732339 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="483ade17-adcc-4ebd-ad88-4afd29d8eb8d" containerName="revoke-keys" Jul 16 21:49:46.732356 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.732348 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c76040f-e10a-4619-9401-90721fd1ad3e" containerName="revoke-keys" Jul 16 21:49:46.732356 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.732357 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c76040f-e10a-4619-9401-90721fd1ad3e" containerName="revoke-keys" Jul 16 21:49:46.732670 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.732377 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c76040f-e10a-4619-9401-90721fd1ad3e" containerName="revoke-keys" Jul 16 21:49:46.732670 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.732385 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c76040f-e10a-4619-9401-90721fd1ad3e" containerName="revoke-keys" Jul 16 21:49:46.732670 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.732401 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b94194ed-41a6-4769-b44a-8b3c21a57935" containerName="revoke-keys" Jul 16 21:49:46.732670 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.732409 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b94194ed-41a6-4769-b44a-8b3c21a57935" containerName="revoke-keys" Jul 16 21:49:46.732670 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.732509 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c76040f-e10a-4619-9401-90721fd1ad3e" containerName="revoke-keys" Jul 16 21:49:46.732670 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.732520 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="483ade17-adcc-4ebd-ad88-4afd29d8eb8d" containerName="revoke-keys" Jul 16 21:49:46.732670 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.732529 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="483ade17-adcc-4ebd-ad88-4afd29d8eb8d" containerName="revoke-keys" Jul 16 21:49:46.732670 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.732543 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b94194ed-41a6-4769-b44a-8b3c21a57935" containerName="revoke-keys" Jul 16 21:49:46.732670 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.732552 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c76040f-e10a-4619-9401-90721fd1ad3e" containerName="revoke-keys" Jul 16 21:49:46.737230 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.737211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" Jul 16 21:49:46.740015 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.739993 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:49:46.743732 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.743711 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn"] Jul 16 21:49:46.792039 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.792007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de677861-8b04-4f89-a8f8-21065816cc4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn\" (UID: \"de677861-8b04-4f89-a8f8-21065816cc4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" Jul 16 21:49:46.893296 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.893259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de677861-8b04-4f89-a8f8-21065816cc4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn\" (UID: \"de677861-8b04-4f89-a8f8-21065816cc4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" Jul 16 21:49:46.893884 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:46.893864 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de677861-8b04-4f89-a8f8-21065816cc4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn\" (UID: \"de677861-8b04-4f89-a8f8-21065816cc4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" Jul 16 21:49:47.049587 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:47.049496 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" Jul 16 21:49:47.183308 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:47.183095 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn"] Jul 16 21:49:47.185541 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:49:47.185517 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde677861_8b04_4f89_a8f8_21065816cc4c.slice/crio-579581c257e24a92005aeae86dfe0b26c8dee6aea86d104de130aae11f78d6c1 WatchSource:0}: Error finding container 579581c257e24a92005aeae86dfe0b26c8dee6aea86d104de130aae11f78d6c1: Status 404 returned error can't find the container with id 579581c257e24a92005aeae86dfe0b26c8dee6aea86d104de130aae11f78d6c1 Jul 16 21:49:47.814001 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:47.813903 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" event={"ID":"de677861-8b04-4f89-a8f8-21065816cc4c","Type":"ContainerStarted","Data":"712495a214399ea11230bb7c3bb54d816614ebbc6e48c33007b352cb2f6e3c6d"} Jul 16 21:49:47.814001 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:47.813995 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" event={"ID":"de677861-8b04-4f89-a8f8-21065816cc4c","Type":"ContainerStarted","Data":"579581c257e24a92005aeae86dfe0b26c8dee6aea86d104de130aae11f78d6c1"} Jul 16 21:49:47.831231 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:47.831167 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" podStartSLOduration=1.831152876 podStartE2EDuration="1.831152876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:47.828925843 +0000 UTC m=+2404.004034038" watchObservedRunningTime="2026-07-16 21:49:47.831152876 +0000 UTC m=+2404.006261099" Jul 16 21:49:48.820561 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:48.820526 2577 generic.go:356] "Generic (PLEG): container finished" podID="de677861-8b04-4f89-a8f8-21065816cc4c" containerID="712495a214399ea11230bb7c3bb54d816614ebbc6e48c33007b352cb2f6e3c6d" exitCode=6 Jul 16 21:49:48.820935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:48.820575 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" event={"ID":"de677861-8b04-4f89-a8f8-21065816cc4c","Type":"ContainerDied","Data":"712495a214399ea11230bb7c3bb54d816614ebbc6e48c33007b352cb2f6e3c6d"} Jul 16 21:49:48.820935 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:48.820796 2577 scope.go:117] "RemoveContainer" containerID="712495a214399ea11230bb7c3bb54d816614ebbc6e48c33007b352cb2f6e3c6d" Jul 16 21:49:49.827443 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:49.827356 2577 generic.go:356] "Generic (PLEG): container finished" podID="de677861-8b04-4f89-a8f8-21065816cc4c" containerID="8e39fc0bc27923dcb3cf774cc86677028cb526e9464e708354e2ced81e7fbe02" exitCode=6 Jul 16 21:49:49.827844 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:49.827445 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" event={"ID":"de677861-8b04-4f89-a8f8-21065816cc4c","Type":"ContainerDied","Data":"8e39fc0bc27923dcb3cf774cc86677028cb526e9464e708354e2ced81e7fbe02"} Jul 16 21:49:49.827844 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:49.827511 2577 scope.go:117] "RemoveContainer" containerID="712495a214399ea11230bb7c3bb54d816614ebbc6e48c33007b352cb2f6e3c6d" Jul 16 21:49:49.827844 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:49.827610 2577 scope.go:117] "RemoveContainer" containerID="8e39fc0bc27923dcb3cf774cc86677028cb526e9464e708354e2ced81e7fbe02" Jul 16 21:49:49.827844 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:49.827812 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn_odh-ai-gateway-infra(de677861-8b04-4f89-a8f8-21065816cc4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" podUID="de677861-8b04-4f89-a8f8-21065816cc4c" Jul 16 21:49:50.450668 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:50.450631 2577 scope.go:117] "RemoveContainer" containerID="1c07562366e45320fefeb92430686b72f3068ed045497950bedf5d5b27cb2daf" Jul 16 21:49:50.834324 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:50.834245 2577 scope.go:117] "RemoveContainer" containerID="8e39fc0bc27923dcb3cf774cc86677028cb526e9464e708354e2ced81e7fbe02" Jul 16 21:49:50.834680 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:50.834419 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn_odh-ai-gateway-infra(de677861-8b04-4f89-a8f8-21065816cc4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" podUID="de677861-8b04-4f89-a8f8-21065816cc4c" Jul 16 21:49:51.843339 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:51.843312 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerID="5e73f7a24cec55f506c938c3d3a1ce8f9022c289ade9d517b9e9db8192befe08" exitCode=1 Jul 16 21:49:51.843663 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:51.843383 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" event={"ID":"9fa7af33-c34c-479b-ad5b-d451bdf6b61b","Type":"ContainerDied","Data":"5e73f7a24cec55f506c938c3d3a1ce8f9022c289ade9d517b9e9db8192befe08"} Jul 16 21:49:51.843663 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:51.843440 2577 scope.go:117] "RemoveContainer" containerID="1c07562366e45320fefeb92430686b72f3068ed045497950bedf5d5b27cb2daf" Jul 16 21:49:51.843779 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:51.843733 2577 scope.go:117] "RemoveContainer" containerID="5e73f7a24cec55f506c938c3d3a1ce8f9022c289ade9d517b9e9db8192befe08" Jul 16 21:49:51.843946 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:51.843926 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:49:54.739100 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:54.739065 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:49:54.739508 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:54.739410 2577 scope.go:117] "RemoveContainer" containerID="5e73f7a24cec55f506c938c3d3a1ce8f9022c289ade9d517b9e9db8192befe08" Jul 16 21:49:54.739600 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:54.739582 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:49:55.594303 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:55.594266 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk"] Jul 16 21:49:55.594618 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:55.594606 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c76040f-e10a-4619-9401-90721fd1ad3e" containerName="revoke-keys" Jul 16 21:49:55.594667 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:55.594619 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c76040f-e10a-4619-9401-90721fd1ad3e" containerName="revoke-keys" Jul 16 21:49:55.594667 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:55.594645 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="483ade17-adcc-4ebd-ad88-4afd29d8eb8d" containerName="revoke-keys" Jul 16 21:49:55.594667 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:55.594650 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="483ade17-adcc-4ebd-ad88-4afd29d8eb8d" containerName="revoke-keys" Jul 16 21:49:55.594762 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:55.594701 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c76040f-e10a-4619-9401-90721fd1ad3e" containerName="revoke-keys" Jul 16 21:49:55.594762 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:55.594710 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="483ade17-adcc-4ebd-ad88-4afd29d8eb8d" containerName="revoke-keys" Jul 16 21:49:55.599134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:55.599117 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" Jul 16 21:49:55.607938 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:55.607912 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk"] Jul 16 21:49:55.670269 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:55.670239 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bdd57df-434d-49ec-a243-6b75df9902e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk\" (UID: \"1bdd57df-434d-49ec-a243-6b75df9902e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" Jul 16 21:49:55.771259 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:55.771233 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bdd57df-434d-49ec-a243-6b75df9902e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk\" (UID: \"1bdd57df-434d-49ec-a243-6b75df9902e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" Jul 16 21:49:55.771912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:55.771891 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bdd57df-434d-49ec-a243-6b75df9902e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk\" (UID: \"1bdd57df-434d-49ec-a243-6b75df9902e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" Jul 16 21:49:55.911307 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:55.911283 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" Jul 16 21:49:56.066475 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:56.066446 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk"] Jul 16 21:49:56.067848 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:49:56.067820 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdd57df_434d_49ec_a243_6b75df9902e8.slice/crio-99f5ebb61f0c3e76500ce4f2b13a6cecc5b42c717c30752392066e7040608498 WatchSource:0}: Error finding container 99f5ebb61f0c3e76500ce4f2b13a6cecc5b42c717c30752392066e7040608498: Status 404 returned error can't find the container with id 99f5ebb61f0c3e76500ce4f2b13a6cecc5b42c717c30752392066e7040608498 Jul 16 21:49:56.872066 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:56.872036 2577 generic.go:356] "Generic (PLEG): container finished" podID="1bdd57df-434d-49ec-a243-6b75df9902e8" containerID="9dbd3f40272bc3c35acc13d2f90280859f40aec20928d00492bd9760fc6dcef8" exitCode=6 Jul 16 21:49:56.872475 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:56.872073 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" event={"ID":"1bdd57df-434d-49ec-a243-6b75df9902e8","Type":"ContainerDied","Data":"9dbd3f40272bc3c35acc13d2f90280859f40aec20928d00492bd9760fc6dcef8"} Jul 16 21:49:56.872475 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:56.872127 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" event={"ID":"1bdd57df-434d-49ec-a243-6b75df9902e8","Type":"ContainerStarted","Data":"99f5ebb61f0c3e76500ce4f2b13a6cecc5b42c717c30752392066e7040608498"} Jul 16 21:49:56.872475 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:56.872388 2577 scope.go:117] "RemoveContainer" containerID="9dbd3f40272bc3c35acc13d2f90280859f40aec20928d00492bd9760fc6dcef8" Jul 16 21:49:57.879216 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:57.879171 2577 generic.go:356] "Generic (PLEG): container finished" podID="1bdd57df-434d-49ec-a243-6b75df9902e8" containerID="54b78ce85a337af59e2a3d2c1be75708ce79d848f2724f11d9abcea3eead842c" exitCode=6 Jul 16 21:49:57.879655 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:57.879254 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" event={"ID":"1bdd57df-434d-49ec-a243-6b75df9902e8","Type":"ContainerDied","Data":"54b78ce85a337af59e2a3d2c1be75708ce79d848f2724f11d9abcea3eead842c"} Jul 16 21:49:57.879655 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:57.879315 2577 scope.go:117] "RemoveContainer" containerID="9dbd3f40272bc3c35acc13d2f90280859f40aec20928d00492bd9760fc6dcef8" Jul 16 21:49:57.879655 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:57.879482 2577 scope.go:117] "RemoveContainer" containerID="54b78ce85a337af59e2a3d2c1be75708ce79d848f2724f11d9abcea3eead842c" Jul 16 21:49:57.879814 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:57.879720 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk_odh-ai-gateway-infra(1bdd57df-434d-49ec-a243-6b75df9902e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" podUID="1bdd57df-434d-49ec-a243-6b75df9902e8" Jul 16 21:49:58.885687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:58.885660 2577 scope.go:117] "RemoveContainer" containerID="54b78ce85a337af59e2a3d2c1be75708ce79d848f2724f11d9abcea3eead842c" Jul 16 21:49:58.886066 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:58.885834 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk_odh-ai-gateway-infra(1bdd57df-434d-49ec-a243-6b75df9902e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" podUID="1bdd57df-434d-49ec-a243-6b75df9902e8" Jul 16 21:49:59.738837 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:59.738802 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:49:59.739249 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:49:59.739232 2577 scope.go:117] "RemoveContainer" containerID="5e73f7a24cec55f506c938c3d3a1ce8f9022c289ade9d517b9e9db8192befe08" Jul 16 21:49:59.739436 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:49:59.739414 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:50:03.445517 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:03.445488 2577 scope.go:117] "RemoveContainer" containerID="8e39fc0bc27923dcb3cf774cc86677028cb526e9464e708354e2ced81e7fbe02" Jul 16 21:50:03.916350 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:03.916317 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" event={"ID":"de677861-8b04-4f89-a8f8-21065816cc4c","Type":"ContainerStarted","Data":"f0f0537567852be024e09653ac049257685085eded2d4979002a02e1ac20c0ea"} Jul 16 21:50:04.923714 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:04.923682 2577 generic.go:356] "Generic (PLEG): container finished" podID="de677861-8b04-4f89-a8f8-21065816cc4c" containerID="f0f0537567852be024e09653ac049257685085eded2d4979002a02e1ac20c0ea" exitCode=6 Jul 16 21:50:04.924156 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:04.923752 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" event={"ID":"de677861-8b04-4f89-a8f8-21065816cc4c","Type":"ContainerDied","Data":"f0f0537567852be024e09653ac049257685085eded2d4979002a02e1ac20c0ea"} Jul 16 21:50:04.924156 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:04.923809 2577 scope.go:117] "RemoveContainer" containerID="8e39fc0bc27923dcb3cf774cc86677028cb526e9464e708354e2ced81e7fbe02" Jul 16 21:50:04.924156 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:04.924107 2577 scope.go:117] "RemoveContainer" containerID="f0f0537567852be024e09653ac049257685085eded2d4979002a02e1ac20c0ea" Jul 16 21:50:04.924359 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:04.924329 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn_odh-ai-gateway-infra(de677861-8b04-4f89-a8f8-21065816cc4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" podUID="de677861-8b04-4f89-a8f8-21065816cc4c" Jul 16 21:50:04.949837 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:04.949810 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn"] Jul 16 21:50:04.984080 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:04.984055 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf"] Jul 16 21:50:04.988612 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:04.988596 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" Jul 16 21:50:04.996489 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:04.996449 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf"] Jul 16 21:50:05.156902 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:05.156872 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9cb03ad-0ac2-45f5-98ea-269c58894c0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf\" (UID: \"d9cb03ad-0ac2-45f5-98ea-269c58894c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" Jul 16 21:50:05.258168 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:05.258083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9cb03ad-0ac2-45f5-98ea-269c58894c0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf\" (UID: \"d9cb03ad-0ac2-45f5-98ea-269c58894c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" Jul 16 21:50:05.258665 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:05.258647 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9cb03ad-0ac2-45f5-98ea-269c58894c0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf\" (UID: \"d9cb03ad-0ac2-45f5-98ea-269c58894c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" Jul 16 21:50:05.299768 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:05.299744 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" Jul 16 21:50:05.630365 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:05.630341 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf"] Jul 16 21:50:05.632147 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:50:05.632117 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9cb03ad_0ac2_45f5_98ea_269c58894c0b.slice/crio-636da525149b71cea39acd669a1a78de7c9afb76d0da48d76263b4693ec1682b WatchSource:0}: Error finding container 636da525149b71cea39acd669a1a78de7c9afb76d0da48d76263b4693ec1682b: Status 404 returned error can't find the container with id 636da525149b71cea39acd669a1a78de7c9afb76d0da48d76263b4693ec1682b Jul 16 21:50:05.931997 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:05.931964 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" event={"ID":"d9cb03ad-0ac2-45f5-98ea-269c58894c0b","Type":"ContainerStarted","Data":"1340b730c0456cf298571405f43f05c0ac84037e8969807847db2d29c6777ebc"} Jul 16 21:50:05.932326 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:05.932021 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" event={"ID":"d9cb03ad-0ac2-45f5-98ea-269c58894c0b","Type":"ContainerStarted","Data":"636da525149b71cea39acd669a1a78de7c9afb76d0da48d76263b4693ec1682b"} Jul 16 21:50:05.962752 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:05.962706 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" podStartSLOduration=1.962691736 podStartE2EDuration="1.962691736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:05.960735789 +0000 UTC m=+2422.135843980" watchObservedRunningTime="2026-07-16 21:50:05.962691736 +0000 UTC m=+2422.137799928" Jul 16 21:50:06.054735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:06.054708 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" Jul 16 21:50:06.165104 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:06.165041 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de677861-8b04-4f89-a8f8-21065816cc4c-maas-api-service-ca\") pod \"de677861-8b04-4f89-a8f8-21065816cc4c\" (UID: \"de677861-8b04-4f89-a8f8-21065816cc4c\") " Jul 16 21:50:06.165364 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:06.165345 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de677861-8b04-4f89-a8f8-21065816cc4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "de677861-8b04-4f89-a8f8-21065816cc4c" (UID: "de677861-8b04-4f89-a8f8-21065816cc4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:50:06.266386 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:06.266364 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de677861-8b04-4f89-a8f8-21065816cc4c-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:50:06.938812 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:06.938780 2577 generic.go:356] "Generic (PLEG): container finished" podID="d9cb03ad-0ac2-45f5-98ea-269c58894c0b" containerID="1340b730c0456cf298571405f43f05c0ac84037e8969807847db2d29c6777ebc" exitCode=6 Jul 16 21:50:06.939266 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:06.938868 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" event={"ID":"d9cb03ad-0ac2-45f5-98ea-269c58894c0b","Type":"ContainerDied","Data":"1340b730c0456cf298571405f43f05c0ac84037e8969807847db2d29c6777ebc"} Jul 16 21:50:06.939266 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:06.939038 2577 scope.go:117] "RemoveContainer" containerID="1340b730c0456cf298571405f43f05c0ac84037e8969807847db2d29c6777ebc" Jul 16 21:50:06.940341 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:06.940323 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" event={"ID":"de677861-8b04-4f89-a8f8-21065816cc4c","Type":"ContainerDied","Data":"579581c257e24a92005aeae86dfe0b26c8dee6aea86d104de130aae11f78d6c1"} Jul 16 21:50:06.940341 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:06.940345 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn" Jul 16 21:50:06.940504 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:06.940368 2577 scope.go:117] "RemoveContainer" containerID="f0f0537567852be024e09653ac049257685085eded2d4979002a02e1ac20c0ea" Jul 16 21:50:06.968399 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:06.968375 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn"] Jul 16 21:50:06.974013 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:06.973982 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-8c7wn"] Jul 16 21:50:07.947867 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:07.947833 2577 generic.go:356] "Generic (PLEG): container finished" podID="d9cb03ad-0ac2-45f5-98ea-269c58894c0b" containerID="96dbc1aaab25c55546c02d9a5940d862ea0f8c12a7b314ba2ec6cdda0f28bbc9" exitCode=6 Jul 16 21:50:07.948307 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:07.947889 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" event={"ID":"d9cb03ad-0ac2-45f5-98ea-269c58894c0b","Type":"ContainerDied","Data":"96dbc1aaab25c55546c02d9a5940d862ea0f8c12a7b314ba2ec6cdda0f28bbc9"} Jul 16 21:50:07.948307 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:07.947941 2577 scope.go:117] "RemoveContainer" containerID="1340b730c0456cf298571405f43f05c0ac84037e8969807847db2d29c6777ebc" Jul 16 21:50:07.948307 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:07.948097 2577 scope.go:117] "RemoveContainer" containerID="96dbc1aaab25c55546c02d9a5940d862ea0f8c12a7b314ba2ec6cdda0f28bbc9" Jul 16 21:50:07.948430 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:07.948329 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf_odh-ai-gateway-infra(d9cb03ad-0ac2-45f5-98ea-269c58894c0b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" podUID="d9cb03ad-0ac2-45f5-98ea-269c58894c0b" Jul 16 21:50:08.450369 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:08.450339 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de677861-8b04-4f89-a8f8-21065816cc4c" path="/var/lib/kubelet/pods/de677861-8b04-4f89-a8f8-21065816cc4c/volumes" Jul 16 21:50:08.954124 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:08.954095 2577 scope.go:117] "RemoveContainer" containerID="96dbc1aaab25c55546c02d9a5940d862ea0f8c12a7b314ba2ec6cdda0f28bbc9" Jul 16 21:50:08.954492 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:08.954320 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf_odh-ai-gateway-infra(d9cb03ad-0ac2-45f5-98ea-269c58894c0b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" podUID="d9cb03ad-0ac2-45f5-98ea-269c58894c0b" Jul 16 21:50:11.445414 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:11.445386 2577 scope.go:117] "RemoveContainer" containerID="5e73f7a24cec55f506c938c3d3a1ce8f9022c289ade9d517b9e9db8192befe08" Jul 16 21:50:11.445885 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:11.445449 2577 scope.go:117] "RemoveContainer" containerID="54b78ce85a337af59e2a3d2c1be75708ce79d848f2724f11d9abcea3eead842c" Jul 16 21:50:11.445885 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:11.445620 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:50:11.973888 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:11.973854 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" event={"ID":"1bdd57df-434d-49ec-a243-6b75df9902e8","Type":"ContainerStarted","Data":"478572ebe00fdedb76dd60f6a5a47dbf242d25df58c6f527defaee65970be9c8"} Jul 16 21:50:11.994848 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:11.994800 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" podStartSLOduration=16.994785663000002 podStartE2EDuration="16.994785663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:11.992913298 +0000 UTC m=+2428.168021491" watchObservedRunningTime="2026-07-16 21:50:11.994785663 +0000 UTC m=+2428.169894020" Jul 16 21:50:12.981384 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:12.981350 2577 generic.go:356] "Generic (PLEG): container finished" podID="1bdd57df-434d-49ec-a243-6b75df9902e8" containerID="478572ebe00fdedb76dd60f6a5a47dbf242d25df58c6f527defaee65970be9c8" exitCode=6 Jul 16 21:50:12.981781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:12.981426 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" event={"ID":"1bdd57df-434d-49ec-a243-6b75df9902e8","Type":"ContainerDied","Data":"478572ebe00fdedb76dd60f6a5a47dbf242d25df58c6f527defaee65970be9c8"} Jul 16 21:50:12.981781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:12.981490 2577 scope.go:117] "RemoveContainer" containerID="54b78ce85a337af59e2a3d2c1be75708ce79d848f2724f11d9abcea3eead842c" Jul 16 21:50:12.981781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:12.981711 2577 scope.go:117] "RemoveContainer" containerID="478572ebe00fdedb76dd60f6a5a47dbf242d25df58c6f527defaee65970be9c8" Jul 16 21:50:12.981910 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:12.981895 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk_odh-ai-gateway-infra(1bdd57df-434d-49ec-a243-6b75df9902e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" podUID="1bdd57df-434d-49ec-a243-6b75df9902e8" Jul 16 21:50:13.007882 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:13.007855 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk"] Jul 16 21:50:13.258931 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:13.258858 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk"] Jul 16 21:50:13.259237 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:13.259224 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de677861-8b04-4f89-a8f8-21065816cc4c" containerName="revoke-keys" Jul 16 21:50:13.259289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:13.259238 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="de677861-8b04-4f89-a8f8-21065816cc4c" containerName="revoke-keys" Jul 16 21:50:13.259289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:13.259251 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de677861-8b04-4f89-a8f8-21065816cc4c" containerName="revoke-keys" Jul 16 21:50:13.259289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:13.259256 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="de677861-8b04-4f89-a8f8-21065816cc4c" containerName="revoke-keys" Jul 16 21:50:13.259407 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:13.259322 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="de677861-8b04-4f89-a8f8-21065816cc4c" containerName="revoke-keys" Jul 16 21:50:13.259407 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:13.259331 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="de677861-8b04-4f89-a8f8-21065816cc4c" containerName="revoke-keys" Jul 16 21:50:13.263475 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:13.263458 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk" Jul 16 21:50:13.287790 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:13.287767 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk"] Jul 16 21:50:13.426525 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:13.426494 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc1be84-fa38-44dd-a2b6-373b5211df6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk\" (UID: \"abc1be84-fa38-44dd-a2b6-373b5211df6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk" Jul 16 21:50:13.527559 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:13.527483 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc1be84-fa38-44dd-a2b6-373b5211df6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk\" (UID: \"abc1be84-fa38-44dd-a2b6-373b5211df6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk" Jul 16 21:50:13.528059 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:13.528041 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc1be84-fa38-44dd-a2b6-373b5211df6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk\" (UID: \"abc1be84-fa38-44dd-a2b6-373b5211df6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk" Jul 16 21:50:13.572550 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:13.572529 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk" Jul 16 21:50:13.702598 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:13.702572 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk"] Jul 16 21:50:13.703115 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:50:13.703093 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc1be84_fa38_44dd_a2b6_373b5211df6d.slice/crio-fc1456eb46e5c02e4e8a1d4ef18efd8863a9a3dc49ea8ef22341ece388d1d373 WatchSource:0}: Error finding container fc1456eb46e5c02e4e8a1d4ef18efd8863a9a3dc49ea8ef22341ece388d1d373: Status 404 returned error can't find the container with id fc1456eb46e5c02e4e8a1d4ef18efd8863a9a3dc49ea8ef22341ece388d1d373 Jul 16 21:50:13.988644 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:13.988618 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk" event={"ID":"abc1be84-fa38-44dd-a2b6-373b5211df6d","Type":"ContainerStarted","Data":"fc1456eb46e5c02e4e8a1d4ef18efd8863a9a3dc49ea8ef22341ece388d1d373"} Jul 16 21:50:14.117033 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:14.117005 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" Jul 16 21:50:14.232943 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:14.232865 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bdd57df-434d-49ec-a243-6b75df9902e8-maas-api-service-ca\") pod \"1bdd57df-434d-49ec-a243-6b75df9902e8\" (UID: \"1bdd57df-434d-49ec-a243-6b75df9902e8\") " Jul 16 21:50:14.233232 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:14.233211 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bdd57df-434d-49ec-a243-6b75df9902e8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1bdd57df-434d-49ec-a243-6b75df9902e8" (UID: "1bdd57df-434d-49ec-a243-6b75df9902e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:50:14.334157 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:14.334131 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bdd57df-434d-49ec-a243-6b75df9902e8-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:50:14.997512 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:14.997472 2577 generic.go:356] "Generic (PLEG): container finished" podID="abc1be84-fa38-44dd-a2b6-373b5211df6d" containerID="36b4b4d281c412ad2a8471c3620f9fadd603bea56e0cb43b9c5bdfe00be2868b" exitCode=7 Jul 16 21:50:14.997960 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:14.997558 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk" event={"ID":"abc1be84-fa38-44dd-a2b6-373b5211df6d","Type":"ContainerDied","Data":"36b4b4d281c412ad2a8471c3620f9fadd603bea56e0cb43b9c5bdfe00be2868b"} Jul 16 21:50:14.997960 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:14.997839 2577 scope.go:117] "RemoveContainer" containerID="36b4b4d281c412ad2a8471c3620f9fadd603bea56e0cb43b9c5bdfe00be2868b" Jul 16 21:50:14.999031 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:14.999008 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" event={"ID":"1bdd57df-434d-49ec-a243-6b75df9902e8","Type":"ContainerDied","Data":"99f5ebb61f0c3e76500ce4f2b13a6cecc5b42c717c30752392066e7040608498"} Jul 16 21:50:14.999031 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:14.999026 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk" Jul 16 21:50:14.999215 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:14.999054 2577 scope.go:117] "RemoveContainer" containerID="478572ebe00fdedb76dd60f6a5a47dbf242d25df58c6f527defaee65970be9c8" Jul 16 21:50:15.032120 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:15.032094 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk"] Jul 16 21:50:15.038443 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:15.038419 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-wvnpk"] Jul 16 21:50:16.007237 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:16.007203 2577 generic.go:356] "Generic (PLEG): container finished" podID="abc1be84-fa38-44dd-a2b6-373b5211df6d" containerID="a884e6c622ec86403d6af4d477ba77c359d70890e0e2b5d57dbd38ee91d65020" exitCode=7 Jul 16 21:50:16.007237 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:16.007240 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk" event={"ID":"abc1be84-fa38-44dd-a2b6-373b5211df6d","Type":"ContainerDied","Data":"a884e6c622ec86403d6af4d477ba77c359d70890e0e2b5d57dbd38ee91d65020"} Jul 16 21:50:16.007811 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:16.007287 2577 scope.go:117] "RemoveContainer" containerID="36b4b4d281c412ad2a8471c3620f9fadd603bea56e0cb43b9c5bdfe00be2868b" Jul 16 21:50:16.007811 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:16.007526 2577 scope.go:117] "RemoveContainer" containerID="a884e6c622ec86403d6af4d477ba77c359d70890e0e2b5d57dbd38ee91d65020" Jul 16 21:50:16.007811 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:16.007763 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk_odh-ai-gateway-infra(abc1be84-fa38-44dd-a2b6-373b5211df6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk" podUID="abc1be84-fa38-44dd-a2b6-373b5211df6d" Jul 16 21:50:16.450751 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:16.450717 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bdd57df-434d-49ec-a243-6b75df9902e8" path="/var/lib/kubelet/pods/1bdd57df-434d-49ec-a243-6b75df9902e8/volumes" Jul 16 21:50:17.013851 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:17.013828 2577 scope.go:117] "RemoveContainer" containerID="a884e6c622ec86403d6af4d477ba77c359d70890e0e2b5d57dbd38ee91d65020" Jul 16 21:50:17.014258 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:17.013996 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk_odh-ai-gateway-infra(abc1be84-fa38-44dd-a2b6-373b5211df6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk" podUID="abc1be84-fa38-44dd-a2b6-373b5211df6d" Jul 16 21:50:20.445381 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:20.445350 2577 scope.go:117] "RemoveContainer" containerID="96dbc1aaab25c55546c02d9a5940d862ea0f8c12a7b314ba2ec6cdda0f28bbc9" Jul 16 21:50:21.034904 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:21.034811 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" event={"ID":"d9cb03ad-0ac2-45f5-98ea-269c58894c0b","Type":"ContainerStarted","Data":"342b01c020ae299e7f4a608322097517585ab29c33bcbfcdd3cd7262889e0aef"} Jul 16 21:50:22.042348 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:22.042315 2577 generic.go:356] "Generic (PLEG): container finished" podID="d9cb03ad-0ac2-45f5-98ea-269c58894c0b" containerID="342b01c020ae299e7f4a608322097517585ab29c33bcbfcdd3cd7262889e0aef" exitCode=6 Jul 16 21:50:22.042809 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:22.042384 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" event={"ID":"d9cb03ad-0ac2-45f5-98ea-269c58894c0b","Type":"ContainerDied","Data":"342b01c020ae299e7f4a608322097517585ab29c33bcbfcdd3cd7262889e0aef"} Jul 16 21:50:22.042809 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:22.042444 2577 scope.go:117] "RemoveContainer" containerID="96dbc1aaab25c55546c02d9a5940d862ea0f8c12a7b314ba2ec6cdda0f28bbc9" Jul 16 21:50:22.042809 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:22.042674 2577 scope.go:117] "RemoveContainer" containerID="342b01c020ae299e7f4a608322097517585ab29c33bcbfcdd3cd7262889e0aef" Jul 16 21:50:22.042941 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:22.042879 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf_odh-ai-gateway-infra(d9cb03ad-0ac2-45f5-98ea-269c58894c0b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" podUID="d9cb03ad-0ac2-45f5-98ea-269c58894c0b" Jul 16 21:50:22.075526 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:22.075494 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf"] Jul 16 21:50:23.181849 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:23.181825 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" Jul 16 21:50:23.207438 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:23.207412 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9cb03ad-0ac2-45f5-98ea-269c58894c0b-maas-api-service-ca\") pod \"d9cb03ad-0ac2-45f5-98ea-269c58894c0b\" (UID: \"d9cb03ad-0ac2-45f5-98ea-269c58894c0b\") " Jul 16 21:50:23.207824 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:23.207799 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9cb03ad-0ac2-45f5-98ea-269c58894c0b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d9cb03ad-0ac2-45f5-98ea-269c58894c0b" (UID: "d9cb03ad-0ac2-45f5-98ea-269c58894c0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:50:23.308726 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:23.308638 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9cb03ad-0ac2-45f5-98ea-269c58894c0b-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:50:23.445014 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:23.444989 2577 scope.go:117] "RemoveContainer" containerID="5e73f7a24cec55f506c938c3d3a1ce8f9022c289ade9d517b9e9db8192befe08" Jul 16 21:50:23.445170 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:23.445152 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:50:24.055173 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:24.055143 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" Jul 16 21:50:24.055368 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:24.055201 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf" event={"ID":"d9cb03ad-0ac2-45f5-98ea-269c58894c0b","Type":"ContainerDied","Data":"636da525149b71cea39acd669a1a78de7c9afb76d0da48d76263b4693ec1682b"} Jul 16 21:50:24.055368 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:24.055264 2577 scope.go:117] "RemoveContainer" containerID="342b01c020ae299e7f4a608322097517585ab29c33bcbfcdd3cd7262889e0aef" Jul 16 21:50:24.079516 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:24.079480 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf"] Jul 16 21:50:24.086819 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:24.086797 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-hxvqf"] Jul 16 21:50:24.451293 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:24.451259 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9cb03ad-0ac2-45f5-98ea-269c58894c0b" path="/var/lib/kubelet/pods/d9cb03ad-0ac2-45f5-98ea-269c58894c0b/volumes" Jul 16 21:50:29.445133 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:29.445097 2577 scope.go:117] "RemoveContainer" containerID="a884e6c622ec86403d6af4d477ba77c359d70890e0e2b5d57dbd38ee91d65020" Jul 16 21:50:30.093046 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:30.093012 2577 generic.go:356] "Generic (PLEG): container finished" podID="abc1be84-fa38-44dd-a2b6-373b5211df6d" containerID="248bfe95b57a6ba532f5d3cd2716f162fd40e3bc5e16531c11c358dc2e84ce4d" exitCode=7 Jul 16 21:50:30.093239 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:30.093058 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk" event={"ID":"abc1be84-fa38-44dd-a2b6-373b5211df6d","Type":"ContainerDied","Data":"248bfe95b57a6ba532f5d3cd2716f162fd40e3bc5e16531c11c358dc2e84ce4d"} Jul 16 21:50:30.093239 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:30.093102 2577 scope.go:117] "RemoveContainer" containerID="a884e6c622ec86403d6af4d477ba77c359d70890e0e2b5d57dbd38ee91d65020" Jul 16 21:50:30.093415 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:30.093395 2577 scope.go:117] "RemoveContainer" containerID="248bfe95b57a6ba532f5d3cd2716f162fd40e3bc5e16531c11c358dc2e84ce4d" Jul 16 21:50:30.093662 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:30.093635 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk_odh-ai-gateway-infra(abc1be84-fa38-44dd-a2b6-373b5211df6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk" podUID="abc1be84-fa38-44dd-a2b6-373b5211df6d" Jul 16 21:50:31.123638 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:31.123610 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk"] Jul 16 21:50:31.258955 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:31.258930 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk" Jul 16 21:50:31.378387 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:31.378362 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc1be84-fa38-44dd-a2b6-373b5211df6d-maas-api-service-ca\") pod \"abc1be84-fa38-44dd-a2b6-373b5211df6d\" (UID: \"abc1be84-fa38-44dd-a2b6-373b5211df6d\") " Jul 16 21:50:31.378740 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:31.378716 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc1be84-fa38-44dd-a2b6-373b5211df6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "abc1be84-fa38-44dd-a2b6-373b5211df6d" (UID: "abc1be84-fa38-44dd-a2b6-373b5211df6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:50:31.479385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:31.479354 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc1be84-fa38-44dd-a2b6-373b5211df6d-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:50:32.107070 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:32.107036 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk" Jul 16 21:50:32.107256 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:32.107036 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk" event={"ID":"abc1be84-fa38-44dd-a2b6-373b5211df6d","Type":"ContainerDied","Data":"fc1456eb46e5c02e4e8a1d4ef18efd8863a9a3dc49ea8ef22341ece388d1d373"} Jul 16 21:50:32.107256 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:32.107232 2577 scope.go:117] "RemoveContainer" containerID="248bfe95b57a6ba532f5d3cd2716f162fd40e3bc5e16531c11c358dc2e84ce4d" Jul 16 21:50:32.129751 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:32.129727 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk"] Jul 16 21:50:32.135298 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:32.135274 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-8lmzk"] Jul 16 21:50:32.451257 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:32.451224 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abc1be84-fa38-44dd-a2b6-373b5211df6d" path="/var/lib/kubelet/pods/abc1be84-fa38-44dd-a2b6-373b5211df6d/volumes" Jul 16 21:50:34.448868 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:34.448838 2577 scope.go:117] "RemoveContainer" containerID="5e73f7a24cec55f506c938c3d3a1ce8f9022c289ade9d517b9e9db8192befe08" Jul 16 21:50:34.449340 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:34.449110 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:50:45.648684 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.648642 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g"] Jul 16 21:50:45.649157 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649136 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de677861-8b04-4f89-a8f8-21065816cc4c" containerName="revoke-keys" Jul 16 21:50:45.649282 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649160 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="de677861-8b04-4f89-a8f8-21065816cc4c" containerName="revoke-keys" Jul 16 21:50:45.649282 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649173 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bdd57df-434d-49ec-a243-6b75df9902e8" containerName="revoke-keys" Jul 16 21:50:45.649282 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649200 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bdd57df-434d-49ec-a243-6b75df9902e8" containerName="revoke-keys" Jul 16 21:50:45.649282 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649231 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc1be84-fa38-44dd-a2b6-373b5211df6d" containerName="revoke-keys" Jul 16 21:50:45.649282 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649241 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc1be84-fa38-44dd-a2b6-373b5211df6d" containerName="revoke-keys" Jul 16 21:50:45.649282 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649253 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bdd57df-434d-49ec-a243-6b75df9902e8" containerName="revoke-keys" Jul 16 21:50:45.649282 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649261 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bdd57df-434d-49ec-a243-6b75df9902e8" containerName="revoke-keys" Jul 16 21:50:45.649282 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649269 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9cb03ad-0ac2-45f5-98ea-269c58894c0b" containerName="revoke-keys" Jul 16 21:50:45.649282 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649277 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9cb03ad-0ac2-45f5-98ea-269c58894c0b" containerName="revoke-keys" Jul 16 21:50:45.649724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649287 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc1be84-fa38-44dd-a2b6-373b5211df6d" containerName="revoke-keys" Jul 16 21:50:45.649724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649295 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc1be84-fa38-44dd-a2b6-373b5211df6d" containerName="revoke-keys" Jul 16 21:50:45.649724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649305 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9cb03ad-0ac2-45f5-98ea-269c58894c0b" containerName="revoke-keys" Jul 16 21:50:45.649724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649314 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9cb03ad-0ac2-45f5-98ea-269c58894c0b" containerName="revoke-keys" Jul 16 21:50:45.649724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649328 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc1be84-fa38-44dd-a2b6-373b5211df6d" containerName="revoke-keys" Jul 16 21:50:45.649724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649336 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc1be84-fa38-44dd-a2b6-373b5211df6d" containerName="revoke-keys" Jul 16 21:50:45.649724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649350 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9cb03ad-0ac2-45f5-98ea-269c58894c0b" containerName="revoke-keys" Jul 16 21:50:45.649724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649357 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9cb03ad-0ac2-45f5-98ea-269c58894c0b" containerName="revoke-keys" Jul 16 21:50:45.649724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649445 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bdd57df-434d-49ec-a243-6b75df9902e8" containerName="revoke-keys" Jul 16 21:50:45.649724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649461 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9cb03ad-0ac2-45f5-98ea-269c58894c0b" containerName="revoke-keys" Jul 16 21:50:45.649724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649469 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc1be84-fa38-44dd-a2b6-373b5211df6d" containerName="revoke-keys" Jul 16 21:50:45.649724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649478 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc1be84-fa38-44dd-a2b6-373b5211df6d" containerName="revoke-keys" Jul 16 21:50:45.649724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649487 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9cb03ad-0ac2-45f5-98ea-269c58894c0b" containerName="revoke-keys" Jul 16 21:50:45.649724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649496 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="de677861-8b04-4f89-a8f8-21065816cc4c" containerName="revoke-keys" Jul 16 21:50:45.649724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.649504 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bdd57df-434d-49ec-a243-6b75df9902e8" containerName="revoke-keys" Jul 16 21:50:45.654356 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.654335 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g" Jul 16 21:50:45.659012 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.658994 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:50:45.668595 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.668573 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g"] Jul 16 21:50:45.804171 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.804138 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g\" (UID: \"3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g" Jul 16 21:50:45.905085 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.904997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g\" (UID: \"3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g" Jul 16 21:50:45.905617 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.905598 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g\" (UID: \"3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g" Jul 16 21:50:45.963901 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:45.963867 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g" Jul 16 21:50:46.090467 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:46.090441 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g"] Jul 16 21:50:46.093135 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:50:46.093101 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbc09f0_4f7c_42d2_8ea4_b06b1b1b1c73.slice/crio-c777efbfa99d10dc80686d117fe98a7ce4c74a77dd1aa5e95ef4deeb85ee80d1 WatchSource:0}: Error finding container c777efbfa99d10dc80686d117fe98a7ce4c74a77dd1aa5e95ef4deeb85ee80d1: Status 404 returned error can't find the container with id c777efbfa99d10dc80686d117fe98a7ce4c74a77dd1aa5e95ef4deeb85ee80d1 Jul 16 21:50:46.180997 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:46.180925 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g" event={"ID":"3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73","Type":"ContainerStarted","Data":"c777efbfa99d10dc80686d117fe98a7ce4c74a77dd1aa5e95ef4deeb85ee80d1"} Jul 16 21:50:46.789705 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:46.789672 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh"] Jul 16 21:50:46.790094 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:46.790043 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bdd57df-434d-49ec-a243-6b75df9902e8" containerName="revoke-keys" Jul 16 21:50:46.790094 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:46.790054 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bdd57df-434d-49ec-a243-6b75df9902e8" containerName="revoke-keys" Jul 16 21:50:46.790165 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:46.790120 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bdd57df-434d-49ec-a243-6b75df9902e8" containerName="revoke-keys" Jul 16 21:50:46.790165 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:46.790131 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc1be84-fa38-44dd-a2b6-373b5211df6d" containerName="revoke-keys" Jul 16 21:50:46.790165 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:46.790138 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9cb03ad-0ac2-45f5-98ea-269c58894c0b" containerName="revoke-keys" Jul 16 21:50:46.793092 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:46.793077 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh" Jul 16 21:50:46.802347 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:46.802318 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh"] Jul 16 21:50:46.912809 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:46.912763 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bffee7d3-567d-4009-802d-60fa9f371b58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh\" (UID: \"bffee7d3-567d-4009-802d-60fa9f371b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh" Jul 16 21:50:47.014015 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:47.013924 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bffee7d3-567d-4009-802d-60fa9f371b58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh\" (UID: \"bffee7d3-567d-4009-802d-60fa9f371b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh" Jul 16 21:50:47.014555 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:47.014534 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bffee7d3-567d-4009-802d-60fa9f371b58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh\" (UID: \"bffee7d3-567d-4009-802d-60fa9f371b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh" Jul 16 21:50:47.104597 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:47.104564 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh" Jul 16 21:50:47.188727 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:47.188693 2577 generic.go:356] "Generic (PLEG): container finished" podID="3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" containerID="2d59c3826cd1ef55fdc4b2026428e8619bc7bc9eff360834250e01f935bed0dd" exitCode=6 Jul 16 21:50:47.188890 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:47.188732 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g" event={"ID":"3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73","Type":"ContainerDied","Data":"2d59c3826cd1ef55fdc4b2026428e8619bc7bc9eff360834250e01f935bed0dd"} Jul 16 21:50:47.189236 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:47.189215 2577 scope.go:117] "RemoveContainer" containerID="2d59c3826cd1ef55fdc4b2026428e8619bc7bc9eff360834250e01f935bed0dd" Jul 16 21:50:47.253276 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:47.253252 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh"] Jul 16 21:50:47.255395 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:50:47.255370 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbffee7d3_567d_4009_802d_60fa9f371b58.slice/crio-f079de65c6272c5eb2cf821067952b9eed0048b725287ee2c698ea199e086dbe WatchSource:0}: Error finding container f079de65c6272c5eb2cf821067952b9eed0048b725287ee2c698ea199e086dbe: Status 404 returned error can't find the container with id f079de65c6272c5eb2cf821067952b9eed0048b725287ee2c698ea199e086dbe Jul 16 21:50:48.195996 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:48.195964 2577 generic.go:356] "Generic (PLEG): container finished" podID="bffee7d3-567d-4009-802d-60fa9f371b58" containerID="e9cd8598bc037a57a9f392719f8d5f83e50eaaaa8a0084130e75bb22cfb73863" exitCode=6 Jul 16 21:50:48.196461 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:48.196022 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh" event={"ID":"bffee7d3-567d-4009-802d-60fa9f371b58","Type":"ContainerDied","Data":"e9cd8598bc037a57a9f392719f8d5f83e50eaaaa8a0084130e75bb22cfb73863"} Jul 16 21:50:48.196461 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:48.196078 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh" event={"ID":"bffee7d3-567d-4009-802d-60fa9f371b58","Type":"ContainerStarted","Data":"f079de65c6272c5eb2cf821067952b9eed0048b725287ee2c698ea199e086dbe"} Jul 16 21:50:48.196461 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:48.196336 2577 scope.go:117] "RemoveContainer" containerID="e9cd8598bc037a57a9f392719f8d5f83e50eaaaa8a0084130e75bb22cfb73863" Jul 16 21:50:48.197986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:48.197957 2577 generic.go:356] "Generic (PLEG): container finished" podID="3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" containerID="bbc841f32acc77fbf267b2b53c927fada5a34888b52cb5db328389b6e97d2045" exitCode=6 Jul 16 21:50:48.198080 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:48.197981 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g" event={"ID":"3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73","Type":"ContainerDied","Data":"bbc841f32acc77fbf267b2b53c927fada5a34888b52cb5db328389b6e97d2045"} Jul 16 21:50:48.198080 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:48.198025 2577 scope.go:117] "RemoveContainer" containerID="2d59c3826cd1ef55fdc4b2026428e8619bc7bc9eff360834250e01f935bed0dd" Jul 16 21:50:48.198193 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:48.198171 2577 scope.go:117] "RemoveContainer" containerID="bbc841f32acc77fbf267b2b53c927fada5a34888b52cb5db328389b6e97d2045" Jul 16 21:50:48.198402 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:48.198381 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g_odh-ai-gateway-infra(3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g" podUID="3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" Jul 16 21:50:49.204819 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:49.204787 2577 generic.go:356] "Generic (PLEG): container finished" podID="bffee7d3-567d-4009-802d-60fa9f371b58" containerID="4eb42804f113ba7b0e6b37379b4052f6e524e263339c4927c3d0a229afbbdf63" exitCode=6 Jul 16 21:50:49.205288 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:49.204870 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh" event={"ID":"bffee7d3-567d-4009-802d-60fa9f371b58","Type":"ContainerDied","Data":"4eb42804f113ba7b0e6b37379b4052f6e524e263339c4927c3d0a229afbbdf63"} Jul 16 21:50:49.205288 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:49.204929 2577 scope.go:117] "RemoveContainer" containerID="e9cd8598bc037a57a9f392719f8d5f83e50eaaaa8a0084130e75bb22cfb73863" Jul 16 21:50:49.205288 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:49.205144 2577 scope.go:117] "RemoveContainer" containerID="4eb42804f113ba7b0e6b37379b4052f6e524e263339c4927c3d0a229afbbdf63" Jul 16 21:50:49.205451 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:49.205417 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh_odh-ai-gateway-infra(bffee7d3-567d-4009-802d-60fa9f371b58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh" podUID="bffee7d3-567d-4009-802d-60fa9f371b58" Jul 16 21:50:49.206670 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:49.206654 2577 scope.go:117] "RemoveContainer" containerID="bbc841f32acc77fbf267b2b53c927fada5a34888b52cb5db328389b6e97d2045" Jul 16 21:50:49.206838 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:49.206822 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g_odh-ai-gateway-infra(3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g" podUID="3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" Jul 16 21:50:49.445834 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:49.445801 2577 scope.go:117] "RemoveContainer" containerID="5e73f7a24cec55f506c938c3d3a1ce8f9022c289ade9d517b9e9db8192befe08" Jul 16 21:50:49.446000 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:49.445963 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:50:50.216545 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:50.216513 2577 scope.go:117] "RemoveContainer" containerID="4eb42804f113ba7b0e6b37379b4052f6e524e263339c4927c3d0a229afbbdf63" Jul 16 21:50:50.217015 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:50.216751 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh_odh-ai-gateway-infra(bffee7d3-567d-4009-802d-60fa9f371b58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh" podUID="bffee7d3-567d-4009-802d-60fa9f371b58" Jul 16 21:50:55.046545 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:55.046510 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b"] Jul 16 21:50:55.049944 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:55.049923 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b" Jul 16 21:50:55.063305 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:55.063276 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b"] Jul 16 21:50:55.186444 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:55.186381 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d79af2-4531-47aa-bf32-d904454cbf76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-sls4b\" (UID: \"f5d79af2-4531-47aa-bf32-d904454cbf76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b" Jul 16 21:50:55.287520 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:55.287488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d79af2-4531-47aa-bf32-d904454cbf76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-sls4b\" (UID: \"f5d79af2-4531-47aa-bf32-d904454cbf76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b" Jul 16 21:50:55.288109 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:55.288091 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d79af2-4531-47aa-bf32-d904454cbf76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-sls4b\" (UID: \"f5d79af2-4531-47aa-bf32-d904454cbf76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b" Jul 16 21:50:55.359157 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:55.359072 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b" Jul 16 21:50:55.495357 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:55.495332 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b"] Jul 16 21:50:55.497291 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:50:55.497266 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d79af2_4531_47aa_bf32_d904454cbf76.slice/crio-4d1a356dc820d11491847f39ead42bf54a20f661efb95e6e1614820aef24fbbd WatchSource:0}: Error finding container 4d1a356dc820d11491847f39ead42bf54a20f661efb95e6e1614820aef24fbbd: Status 404 returned error can't find the container with id 4d1a356dc820d11491847f39ead42bf54a20f661efb95e6e1614820aef24fbbd Jul 16 21:50:56.249459 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:56.249428 2577 generic.go:356] "Generic (PLEG): container finished" podID="f5d79af2-4531-47aa-bf32-d904454cbf76" containerID="8c97d8fd5c7dbed4c7f1c951ca8b3530a89745463a5027dd6f9e748b5af3e27e" exitCode=6 Jul 16 21:50:56.249781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:56.249481 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b" event={"ID":"f5d79af2-4531-47aa-bf32-d904454cbf76","Type":"ContainerDied","Data":"8c97d8fd5c7dbed4c7f1c951ca8b3530a89745463a5027dd6f9e748b5af3e27e"} Jul 16 21:50:56.249781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:56.249534 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b" event={"ID":"f5d79af2-4531-47aa-bf32-d904454cbf76","Type":"ContainerStarted","Data":"4d1a356dc820d11491847f39ead42bf54a20f661efb95e6e1614820aef24fbbd"} Jul 16 21:50:56.249781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:56.249715 2577 scope.go:117] "RemoveContainer" containerID="8c97d8fd5c7dbed4c7f1c951ca8b3530a89745463a5027dd6f9e748b5af3e27e" Jul 16 21:50:57.256387 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:57.256308 2577 generic.go:356] "Generic (PLEG): container finished" podID="f5d79af2-4531-47aa-bf32-d904454cbf76" containerID="98a954b3e38479463eac48eaf71a047740fb8791140f761e634b3c7e98054d8d" exitCode=6 Jul 16 21:50:57.256762 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:57.256392 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b" event={"ID":"f5d79af2-4531-47aa-bf32-d904454cbf76","Type":"ContainerDied","Data":"98a954b3e38479463eac48eaf71a047740fb8791140f761e634b3c7e98054d8d"} Jul 16 21:50:57.256762 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:57.256446 2577 scope.go:117] "RemoveContainer" containerID="8c97d8fd5c7dbed4c7f1c951ca8b3530a89745463a5027dd6f9e748b5af3e27e" Jul 16 21:50:57.256762 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:57.256565 2577 scope.go:117] "RemoveContainer" containerID="98a954b3e38479463eac48eaf71a047740fb8791140f761e634b3c7e98054d8d" Jul 16 21:50:57.256896 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:57.256804 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-sls4b_odh-ai-gateway-infra(f5d79af2-4531-47aa-bf32-d904454cbf76)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b" podUID="f5d79af2-4531-47aa-bf32-d904454cbf76" Jul 16 21:50:58.269727 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:50:58.269699 2577 scope.go:117] "RemoveContainer" containerID="98a954b3e38479463eac48eaf71a047740fb8791140f761e634b3c7e98054d8d" Jul 16 21:50:58.270100 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:50:58.269871 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-sls4b_odh-ai-gateway-infra(f5d79af2-4531-47aa-bf32-d904454cbf76)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b" podUID="f5d79af2-4531-47aa-bf32-d904454cbf76" Jul 16 21:51:02.445255 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:02.445222 2577 scope.go:117] "RemoveContainer" containerID="5e73f7a24cec55f506c938c3d3a1ce8f9022c289ade9d517b9e9db8192befe08" Jul 16 21:51:02.445650 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:02.445427 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:51:03.305449 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:03.305413 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc"] Jul 16 21:51:03.311605 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:03.311576 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" Jul 16 21:51:03.324540 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:03.324514 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc"] Jul 16 21:51:03.445329 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:03.445299 2577 scope.go:117] "RemoveContainer" containerID="bbc841f32acc77fbf267b2b53c927fada5a34888b52cb5db328389b6e97d2045" Jul 16 21:51:03.463940 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:03.463911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f713ae78-8479-4e91-a40d-c1320b6b88c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc\" (UID: \"f713ae78-8479-4e91-a40d-c1320b6b88c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" Jul 16 21:51:03.564760 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:03.564676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f713ae78-8479-4e91-a40d-c1320b6b88c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc\" (UID: \"f713ae78-8479-4e91-a40d-c1320b6b88c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" Jul 16 21:51:03.565370 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:03.565350 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f713ae78-8479-4e91-a40d-c1320b6b88c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc\" (UID: \"f713ae78-8479-4e91-a40d-c1320b6b88c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" Jul 16 21:51:03.621460 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:03.621433 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" Jul 16 21:51:03.756212 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:03.756159 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc"] Jul 16 21:51:03.757762 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:51:03.757738 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf713ae78_8479_4e91_a40d_c1320b6b88c1.slice/crio-f9b6b8afbb383e73afbf35f58aa0ebe1c62f734473ad17432da41f5891569538 WatchSource:0}: Error finding container f9b6b8afbb383e73afbf35f58aa0ebe1c62f734473ad17432da41f5891569538: Status 404 returned error can't find the container with id f9b6b8afbb383e73afbf35f58aa0ebe1c62f734473ad17432da41f5891569538 Jul 16 21:51:04.305170 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:04.305091 2577 generic.go:356] "Generic (PLEG): container finished" podID="3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" containerID="4afce9a6927af591f0a979ebdbe66c034c27b1080b353d28b43eb0fa6230d5db" exitCode=6 Jul 16 21:51:04.305170 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:04.305159 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g" event={"ID":"3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73","Type":"ContainerDied","Data":"4afce9a6927af591f0a979ebdbe66c034c27b1080b353d28b43eb0fa6230d5db"} Jul 16 21:51:04.305387 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:04.305236 2577 scope.go:117] "RemoveContainer" containerID="bbc841f32acc77fbf267b2b53c927fada5a34888b52cb5db328389b6e97d2045" Jul 16 21:51:04.305521 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:04.305505 2577 scope.go:117] "RemoveContainer" containerID="4afce9a6927af591f0a979ebdbe66c034c27b1080b353d28b43eb0fa6230d5db" Jul 16 21:51:04.305775 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:04.305751 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g_odh-ai-gateway-infra(3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g" podUID="3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" Jul 16 21:51:04.306856 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:04.306837 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" event={"ID":"f713ae78-8479-4e91-a40d-c1320b6b88c1","Type":"ContainerStarted","Data":"08806d8b79c0ced44c993e33b8eb3641984ed07e9723f1656cc8397888083182"} Jul 16 21:51:04.306943 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:04.306871 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" event={"ID":"f713ae78-8479-4e91-a40d-c1320b6b88c1","Type":"ContainerStarted","Data":"f9b6b8afbb383e73afbf35f58aa0ebe1c62f734473ad17432da41f5891569538"} Jul 16 21:51:04.346169 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:04.346123 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" podStartSLOduration=1.3461088540000001 podStartE2EDuration="1.346108854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:04.344509862 +0000 UTC m=+2480.519618064" watchObservedRunningTime="2026-07-16 21:51:04.346108854 +0000 UTC m=+2480.521217044" Jul 16 21:51:05.314336 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:05.314302 2577 generic.go:356] "Generic (PLEG): container finished" podID="f713ae78-8479-4e91-a40d-c1320b6b88c1" containerID="08806d8b79c0ced44c993e33b8eb3641984ed07e9723f1656cc8397888083182" exitCode=7 Jul 16 21:51:05.314804 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:05.314352 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" event={"ID":"f713ae78-8479-4e91-a40d-c1320b6b88c1","Type":"ContainerDied","Data":"08806d8b79c0ced44c993e33b8eb3641984ed07e9723f1656cc8397888083182"} Jul 16 21:51:05.314804 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:05.314593 2577 scope.go:117] "RemoveContainer" containerID="08806d8b79c0ced44c993e33b8eb3641984ed07e9723f1656cc8397888083182" Jul 16 21:51:05.343026 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:05.342998 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g"] Jul 16 21:51:05.445391 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:05.445366 2577 scope.go:117] "RemoveContainer" containerID="4eb42804f113ba7b0e6b37379b4052f6e524e263339c4927c3d0a229afbbdf63" Jul 16 21:51:05.497155 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:05.497135 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g" Jul 16 21:51:05.581278 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:05.581212 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73-maas-api-service-ca\") pod \"3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73\" (UID: \"3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73\") " Jul 16 21:51:05.581563 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:05.581538 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" (UID: "3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:51:05.682564 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:05.682537 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:51:06.320874 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:06.320797 2577 generic.go:356] "Generic (PLEG): container finished" podID="bffee7d3-567d-4009-802d-60fa9f371b58" containerID="b0baed0d5994a82274d000fbd30c45eb0769bff6e7aa628e9f09c2683ab33cd4" exitCode=6 Jul 16 21:51:06.321314 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:06.320874 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh" event={"ID":"bffee7d3-567d-4009-802d-60fa9f371b58","Type":"ContainerDied","Data":"b0baed0d5994a82274d000fbd30c45eb0769bff6e7aa628e9f09c2683ab33cd4"} Jul 16 21:51:06.321314 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:06.320933 2577 scope.go:117] "RemoveContainer" containerID="4eb42804f113ba7b0e6b37379b4052f6e524e263339c4927c3d0a229afbbdf63" Jul 16 21:51:06.321314 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:06.321211 2577 scope.go:117] "RemoveContainer" containerID="b0baed0d5994a82274d000fbd30c45eb0769bff6e7aa628e9f09c2683ab33cd4" Jul 16 21:51:06.321489 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:06.321459 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh_odh-ai-gateway-infra(bffee7d3-567d-4009-802d-60fa9f371b58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh" podUID="bffee7d3-567d-4009-802d-60fa9f371b58" Jul 16 21:51:06.322344 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:06.322325 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g" Jul 16 21:51:06.322417 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:06.322349 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g" event={"ID":"3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73","Type":"ContainerDied","Data":"c777efbfa99d10dc80686d117fe98a7ce4c74a77dd1aa5e95ef4deeb85ee80d1"} Jul 16 21:51:06.324120 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:06.324096 2577 generic.go:356] "Generic (PLEG): container finished" podID="f713ae78-8479-4e91-a40d-c1320b6b88c1" containerID="78ac737fec296d987c154679e6581909ae1b4855d1d7fd1f3c1d8991a431766a" exitCode=7 Jul 16 21:51:06.324221 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:06.324169 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" event={"ID":"f713ae78-8479-4e91-a40d-c1320b6b88c1","Type":"ContainerDied","Data":"78ac737fec296d987c154679e6581909ae1b4855d1d7fd1f3c1d8991a431766a"} Jul 16 21:51:06.324325 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:06.324313 2577 scope.go:117] "RemoveContainer" containerID="78ac737fec296d987c154679e6581909ae1b4855d1d7fd1f3c1d8991a431766a" Jul 16 21:51:06.324496 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:06.324478 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc_odh-ai-gateway-infra(f713ae78-8479-4e91-a40d-c1320b6b88c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" podUID="f713ae78-8479-4e91-a40d-c1320b6b88c1" Jul 16 21:51:06.332710 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:06.332693 2577 scope.go:117] "RemoveContainer" containerID="4afce9a6927af591f0a979ebdbe66c034c27b1080b353d28b43eb0fa6230d5db" Jul 16 21:51:06.340178 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:06.340162 2577 scope.go:117] "RemoveContainer" containerID="08806d8b79c0ced44c993e33b8eb3641984ed07e9723f1656cc8397888083182" Jul 16 21:51:06.383867 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:06.383844 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g"] Jul 16 21:51:06.389011 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:06.388989 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lcm6g"] Jul 16 21:51:06.451091 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:06.451063 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" path="/var/lib/kubelet/pods/3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73/volumes" Jul 16 21:51:07.332487 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:07.332458 2577 scope.go:117] "RemoveContainer" containerID="78ac737fec296d987c154679e6581909ae1b4855d1d7fd1f3c1d8991a431766a" Jul 16 21:51:07.332930 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:07.332689 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc_odh-ai-gateway-infra(f713ae78-8479-4e91-a40d-c1320b6b88c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" podUID="f713ae78-8479-4e91-a40d-c1320b6b88c1" Jul 16 21:51:07.361114 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:07.361080 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh"] Jul 16 21:51:07.491243 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:07.491221 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh" Jul 16 21:51:07.500104 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:07.500085 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bffee7d3-567d-4009-802d-60fa9f371b58-maas-api-service-ca\") pod \"bffee7d3-567d-4009-802d-60fa9f371b58\" (UID: \"bffee7d3-567d-4009-802d-60fa9f371b58\") " Jul 16 21:51:07.500426 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:07.500408 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bffee7d3-567d-4009-802d-60fa9f371b58-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bffee7d3-567d-4009-802d-60fa9f371b58" (UID: "bffee7d3-567d-4009-802d-60fa9f371b58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:51:07.601460 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:07.601379 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bffee7d3-567d-4009-802d-60fa9f371b58-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:51:08.340932 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:08.340897 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh" event={"ID":"bffee7d3-567d-4009-802d-60fa9f371b58","Type":"ContainerDied","Data":"f079de65c6272c5eb2cf821067952b9eed0048b725287ee2c698ea199e086dbe"} Jul 16 21:51:08.340932 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:08.340918 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh" Jul 16 21:51:08.341481 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:08.340966 2577 scope.go:117] "RemoveContainer" containerID="b0baed0d5994a82274d000fbd30c45eb0769bff6e7aa628e9f09c2683ab33cd4" Jul 16 21:51:08.369837 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:08.369806 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh"] Jul 16 21:51:08.380667 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:08.380640 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-4dsfh"] Jul 16 21:51:08.450280 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:08.450252 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bffee7d3-567d-4009-802d-60fa9f371b58" path="/var/lib/kubelet/pods/bffee7d3-567d-4009-802d-60fa9f371b58/volumes" Jul 16 21:51:13.445610 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:13.445583 2577 scope.go:117] "RemoveContainer" containerID="98a954b3e38479463eac48eaf71a047740fb8791140f761e634b3c7e98054d8d" Jul 16 21:51:14.372801 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:14.372722 2577 generic.go:356] "Generic (PLEG): container finished" podID="f5d79af2-4531-47aa-bf32-d904454cbf76" containerID="04e148caaa4b20d24355f771f18043b9e25996d98c311e91fb516b71320c1442" exitCode=6 Jul 16 21:51:14.372801 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:14.372768 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b" event={"ID":"f5d79af2-4531-47aa-bf32-d904454cbf76","Type":"ContainerDied","Data":"04e148caaa4b20d24355f771f18043b9e25996d98c311e91fb516b71320c1442"} Jul 16 21:51:14.372979 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:14.372822 2577 scope.go:117] "RemoveContainer" containerID="98a954b3e38479463eac48eaf71a047740fb8791140f761e634b3c7e98054d8d" Jul 16 21:51:14.373082 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:14.373064 2577 scope.go:117] "RemoveContainer" containerID="04e148caaa4b20d24355f771f18043b9e25996d98c311e91fb516b71320c1442" Jul 16 21:51:14.373351 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:14.373324 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-sls4b_odh-ai-gateway-infra(f5d79af2-4531-47aa-bf32-d904454cbf76)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b" podUID="f5d79af2-4531-47aa-bf32-d904454cbf76" Jul 16 21:51:15.410519 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:15.410482 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b"] Jul 16 21:51:15.536401 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:15.536377 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b" Jul 16 21:51:15.568270 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:15.568241 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d79af2-4531-47aa-bf32-d904454cbf76-maas-api-service-ca\") pod \"f5d79af2-4531-47aa-bf32-d904454cbf76\" (UID: \"f5d79af2-4531-47aa-bf32-d904454cbf76\") " Jul 16 21:51:15.568552 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:15.568533 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d79af2-4531-47aa-bf32-d904454cbf76-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5d79af2-4531-47aa-bf32-d904454cbf76" (UID: "f5d79af2-4531-47aa-bf32-d904454cbf76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:51:15.669808 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:15.669734 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d79af2-4531-47aa-bf32-d904454cbf76-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:51:16.384560 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:16.384524 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b" event={"ID":"f5d79af2-4531-47aa-bf32-d904454cbf76","Type":"ContainerDied","Data":"4d1a356dc820d11491847f39ead42bf54a20f661efb95e6e1614820aef24fbbd"} Jul 16 21:51:16.384746 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:16.384577 2577 scope.go:117] "RemoveContainer" containerID="04e148caaa4b20d24355f771f18043b9e25996d98c311e91fb516b71320c1442" Jul 16 21:51:16.384746 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:16.384533 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b" Jul 16 21:51:16.412164 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:16.412141 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b"] Jul 16 21:51:16.414105 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:16.414086 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-sls4b"] Jul 16 21:51:16.450662 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:16.450634 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5d79af2-4531-47aa-bf32-d904454cbf76" path="/var/lib/kubelet/pods/f5d79af2-4531-47aa-bf32-d904454cbf76/volumes" Jul 16 21:51:17.445451 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:17.445422 2577 scope.go:117] "RemoveContainer" containerID="5e73f7a24cec55f506c938c3d3a1ce8f9022c289ade9d517b9e9db8192befe08" Jul 16 21:51:18.397650 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:18.397610 2577 generic.go:356] "Generic (PLEG): container finished" podID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerID="1ff72c9f1483c4b6a53b8e2acb708543a51cc718c8904a814d031500741e973a" exitCode=1 Jul 16 21:51:18.397844 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:18.397660 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" event={"ID":"9fa7af33-c34c-479b-ad5b-d451bdf6b61b","Type":"ContainerDied","Data":"1ff72c9f1483c4b6a53b8e2acb708543a51cc718c8904a814d031500741e973a"} Jul 16 21:51:18.397844 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:18.397760 2577 scope.go:117] "RemoveContainer" containerID="5e73f7a24cec55f506c938c3d3a1ce8f9022c289ade9d517b9e9db8192befe08" Jul 16 21:51:18.398088 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:18.398066 2577 scope.go:117] "RemoveContainer" containerID="1ff72c9f1483c4b6a53b8e2acb708543a51cc718c8904a814d031500741e973a" Jul 16 21:51:18.398299 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:18.398274 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:51:19.445472 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:19.445444 2577 scope.go:117] "RemoveContainer" containerID="78ac737fec296d987c154679e6581909ae1b4855d1d7fd1f3c1d8991a431766a" Jul 16 21:51:19.738693 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:19.738627 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:51:19.738693 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:19.738673 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:51:19.738933 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:19.738922 2577 scope.go:117] "RemoveContainer" containerID="1ff72c9f1483c4b6a53b8e2acb708543a51cc718c8904a814d031500741e973a" Jul 16 21:51:19.739102 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:19.739085 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp_odh-ai-gateway-infra(9fa7af33-c34c-479b-ad5b-d451bdf6b61b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" Jul 16 21:51:20.412409 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:20.412375 2577 generic.go:356] "Generic (PLEG): container finished" podID="f713ae78-8479-4e91-a40d-c1320b6b88c1" containerID="6cf5a2d371c2518239e1ea8ab70302b4bc5098f9430324fbb8865258b4605610" exitCode=7 Jul 16 21:51:20.412577 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:20.412447 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" event={"ID":"f713ae78-8479-4e91-a40d-c1320b6b88c1","Type":"ContainerDied","Data":"6cf5a2d371c2518239e1ea8ab70302b4bc5098f9430324fbb8865258b4605610"} Jul 16 21:51:20.412577 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:20.412506 2577 scope.go:117] "RemoveContainer" containerID="78ac737fec296d987c154679e6581909ae1b4855d1d7fd1f3c1d8991a431766a" Jul 16 21:51:20.412791 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:20.412771 2577 scope.go:117] "RemoveContainer" containerID="6cf5a2d371c2518239e1ea8ab70302b4bc5098f9430324fbb8865258b4605610" Jul 16 21:51:20.412998 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:20.412974 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc_odh-ai-gateway-infra(f713ae78-8479-4e91-a40d-c1320b6b88c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" podUID="f713ae78-8479-4e91-a40d-c1320b6b88c1" Jul 16 21:51:21.452228 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:21.452197 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc"] Jul 16 21:51:21.579761 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:21.579739 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" Jul 16 21:51:21.620390 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:21.620362 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f713ae78-8479-4e91-a40d-c1320b6b88c1-maas-api-service-ca\") pod \"f713ae78-8479-4e91-a40d-c1320b6b88c1\" (UID: \"f713ae78-8479-4e91-a40d-c1320b6b88c1\") " Jul 16 21:51:21.620930 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:21.620886 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f713ae78-8479-4e91-a40d-c1320b6b88c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f713ae78-8479-4e91-a40d-c1320b6b88c1" (UID: "f713ae78-8479-4e91-a40d-c1320b6b88c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:51:21.721805 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:21.721736 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f713ae78-8479-4e91-a40d-c1320b6b88c1-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:51:22.428047 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:22.428017 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" Jul 16 21:51:22.428252 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:22.428063 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc" event={"ID":"f713ae78-8479-4e91-a40d-c1320b6b88c1","Type":"ContainerDied","Data":"f9b6b8afbb383e73afbf35f58aa0ebe1c62f734473ad17432da41f5891569538"} Jul 16 21:51:22.428252 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:22.428122 2577 scope.go:117] "RemoveContainer" containerID="6cf5a2d371c2518239e1ea8ab70302b4bc5098f9430324fbb8865258b4605610" Jul 16 21:51:22.455229 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:22.455178 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc"] Jul 16 21:51:22.463818 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:22.463794 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-f4rqc"] Jul 16 21:51:24.449617 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:24.449583 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f713ae78-8479-4e91-a40d-c1320b6b88c1" path="/var/lib/kubelet/pods/f713ae78-8479-4e91-a40d-c1320b6b88c1/volumes" Jul 16 21:51:29.006379 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:29.006346 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp"] Jul 16 21:51:29.137286 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:29.137258 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:51:29.282975 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:29.282892 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-maas-api-tls\") pod \"9fa7af33-c34c-479b-ad5b-d451bdf6b61b\" (UID: \"9fa7af33-c34c-479b-ad5b-d451bdf6b61b\") " Jul 16 21:51:29.283131 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:29.282992 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntrff\" (UniqueName: \"kubernetes.io/projected/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-kube-api-access-ntrff\") pod \"9fa7af33-c34c-479b-ad5b-d451bdf6b61b\" (UID: \"9fa7af33-c34c-479b-ad5b-d451bdf6b61b\") " Jul 16 21:51:29.283131 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:29.283018 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-openshift-service-ca\") pod \"9fa7af33-c34c-479b-ad5b-d451bdf6b61b\" (UID: \"9fa7af33-c34c-479b-ad5b-d451bdf6b61b\") " Jul 16 21:51:29.283468 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:29.283440 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9fa7af33-c34c-479b-ad5b-d451bdf6b61b" (UID: "9fa7af33-c34c-479b-ad5b-d451bdf6b61b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:51:29.285073 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:29.285047 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9fa7af33-c34c-479b-ad5b-d451bdf6b61b" (UID: "9fa7af33-c34c-479b-ad5b-d451bdf6b61b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:51:29.285216 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:29.285149 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-kube-api-access-ntrff" (OuterVolumeSpecName: "kube-api-access-ntrff") pod "9fa7af33-c34c-479b-ad5b-d451bdf6b61b" (UID: "9fa7af33-c34c-479b-ad5b-d451bdf6b61b"). InnerVolumeSpecName "kube-api-access-ntrff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:51:29.384379 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:29.384347 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntrff\" (UniqueName: \"kubernetes.io/projected/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-kube-api-access-ntrff\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:51:29.384379 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:29.384375 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-openshift-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:51:29.384379 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:29.384386 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fa7af33-c34c-479b-ad5b-d451bdf6b61b-maas-api-tls\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:51:29.470850 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:29.470818 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" Jul 16 21:51:29.471010 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:29.470845 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp" event={"ID":"9fa7af33-c34c-479b-ad5b-d451bdf6b61b","Type":"ContainerDied","Data":"031a42aba7a24998661fae38857df37b0c254e23fd6c3a39d91c30adb84aa9ae"} Jul 16 21:51:29.471010 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:29.470933 2577 scope.go:117] "RemoveContainer" containerID="1ff72c9f1483c4b6a53b8e2acb708543a51cc718c8904a814d031500741e973a" Jul 16 21:51:29.500833 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:29.500798 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp"] Jul 16 21:51:29.505141 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:29.505121 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c1cf683e-6788d9f4dc-vb6kp"] Jul 16 21:51:30.450349 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:30.450317 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" path="/var/lib/kubelet/pods/9fa7af33-c34c-479b-ad5b-d451bdf6b61b/volumes" Jul 16 21:51:45.716202 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716160 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g"] Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716514 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bffee7d3-567d-4009-802d-60fa9f371b58" containerName="revoke-keys" Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716524 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bffee7d3-567d-4009-802d-60fa9f371b58" containerName="revoke-keys" Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716533 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f713ae78-8479-4e91-a40d-c1320b6b88c1" containerName="revoke-keys" Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716539 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f713ae78-8479-4e91-a40d-c1320b6b88c1" containerName="revoke-keys" Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716545 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f713ae78-8479-4e91-a40d-c1320b6b88c1" containerName="revoke-keys" Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716551 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f713ae78-8479-4e91-a40d-c1320b6b88c1" containerName="revoke-keys" Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716562 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716567 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716579 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716584 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716589 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" containerName="revoke-keys" Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716595 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" containerName="revoke-keys" Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716600 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d79af2-4531-47aa-bf32-d904454cbf76" containerName="revoke-keys" Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716604 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d79af2-4531-47aa-bf32-d904454cbf76" containerName="revoke-keys" Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716609 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716614 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:45.716609 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716621 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716627 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716632 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716638 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716648 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d79af2-4531-47aa-bf32-d904454cbf76" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716653 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d79af2-4531-47aa-bf32-d904454cbf76" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716661 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d79af2-4531-47aa-bf32-d904454cbf76" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716666 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d79af2-4531-47aa-bf32-d904454cbf76" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716673 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bffee7d3-567d-4009-802d-60fa9f371b58" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716678 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bffee7d3-567d-4009-802d-60fa9f371b58" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716727 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716734 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716739 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716743 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bffee7d3-567d-4009-802d-60fa9f371b58" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716748 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d79af2-4531-47aa-bf32-d904454cbf76" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716754 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d79af2-4531-47aa-bf32-d904454cbf76" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716759 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716763 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716768 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bffee7d3-567d-4009-802d-60fa9f371b58" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716773 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bffee7d3-567d-4009-802d-60fa9f371b58" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716779 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f713ae78-8479-4e91-a40d-c1320b6b88c1" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716783 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" containerName="revoke-keys" Jul 16 21:51:45.717174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.716788 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f713ae78-8479-4e91-a40d-c1320b6b88c1" containerName="revoke-keys" Jul 16 21:51:45.721365 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.721347 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" Jul 16 21:51:45.724699 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.724674 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:51:45.724805 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.724720 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:51:45.738706 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.738679 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g"] Jul 16 21:51:45.819992 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.819956 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/484e5b37-973b-4ca4-bfa3-399e9e60f196-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g\" (UID: \"484e5b37-973b-4ca4-bfa3-399e9e60f196\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" Jul 16 21:51:45.920854 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.920813 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/484e5b37-973b-4ca4-bfa3-399e9e60f196-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g\" (UID: \"484e5b37-973b-4ca4-bfa3-399e9e60f196\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" Jul 16 21:51:45.921528 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:45.921511 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/484e5b37-973b-4ca4-bfa3-399e9e60f196-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g\" (UID: \"484e5b37-973b-4ca4-bfa3-399e9e60f196\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" Jul 16 21:51:46.031621 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.031537 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" Jul 16 21:51:46.183587 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.183562 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g"] Jul 16 21:51:46.185584 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:51:46.185554 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484e5b37_973b_4ca4_bfa3_399e9e60f196.slice/crio-ecec805b9de53f78140037a8fad7c21860127ef66cfa2ea4667eab90837bded2 WatchSource:0}: Error finding container ecec805b9de53f78140037a8fad7c21860127ef66cfa2ea4667eab90837bded2: Status 404 returned error can't find the container with id ecec805b9de53f78140037a8fad7c21860127ef66cfa2ea4667eab90837bded2 Jul 16 21:51:46.554390 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.554306 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" event={"ID":"484e5b37-973b-4ca4-bfa3-399e9e60f196","Type":"ContainerStarted","Data":"7cfabc209b0c724fd17b741f85d984d391cfb9f3dad8bebc3b7c0b7a228bf73b"} Jul 16 21:51:46.554390 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.554358 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" event={"ID":"484e5b37-973b-4ca4-bfa3-399e9e60f196","Type":"ContainerStarted","Data":"ecec805b9de53f78140037a8fad7c21860127ef66cfa2ea4667eab90837bded2"} Jul 16 21:51:46.584542 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.584498 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" podStartSLOduration=1.584483923 podStartE2EDuration="1.584483923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:46.58321968 +0000 UTC m=+2522.758327863" watchObservedRunningTime="2026-07-16 21:51:46.584483923 +0000 UTC m=+2522.759592114" Jul 16 21:51:46.885331 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.885299 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6"] Jul 16 21:51:46.891535 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.891499 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f713ae78-8479-4e91-a40d-c1320b6b88c1" containerName="revoke-keys" Jul 16 21:51:46.891535 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.891539 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f713ae78-8479-4e91-a40d-c1320b6b88c1" containerName="revoke-keys" Jul 16 21:51:46.891727 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.891566 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:46.891727 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.891572 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:46.891727 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.891580 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:46.891727 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.891586 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:46.891727 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.891594 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" containerName="revoke-keys" Jul 16 21:51:46.891727 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.891603 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cbc09f0-4f7c-42d2-8ea4-b06b1b1b1c73" containerName="revoke-keys" Jul 16 21:51:46.891727 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.891624 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bffee7d3-567d-4009-802d-60fa9f371b58" containerName="revoke-keys" Jul 16 21:51:46.891727 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.891629 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bffee7d3-567d-4009-802d-60fa9f371b58" containerName="revoke-keys" Jul 16 21:51:46.891978 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.891829 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f713ae78-8479-4e91-a40d-c1320b6b88c1" containerName="revoke-keys" Jul 16 21:51:46.891978 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.891845 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:46.891978 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.891851 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:46.891978 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.891857 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fa7af33-c34c-479b-ad5b-d451bdf6b61b" containerName="maas-api" Jul 16 21:51:46.891978 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.891863 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d79af2-4531-47aa-bf32-d904454cbf76" containerName="revoke-keys" Jul 16 21:51:46.894863 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.894846 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6" Jul 16 21:51:46.913506 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:46.913481 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6"] Jul 16 21:51:47.030489 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:47.030445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d8967d-45cb-455f-b2b6-26d8034e70af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6\" (UID: \"20d8967d-45cb-455f-b2b6-26d8034e70af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6" Jul 16 21:51:47.131721 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:47.131683 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d8967d-45cb-455f-b2b6-26d8034e70af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6\" (UID: \"20d8967d-45cb-455f-b2b6-26d8034e70af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6" Jul 16 21:51:47.132320 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:47.132303 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d8967d-45cb-455f-b2b6-26d8034e70af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6\" (UID: \"20d8967d-45cb-455f-b2b6-26d8034e70af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6" Jul 16 21:51:47.204499 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:47.204425 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6" Jul 16 21:51:47.352587 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:47.352558 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6"] Jul 16 21:51:47.355438 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:51:47.355408 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d8967d_45cb_455f_b2b6_26d8034e70af.slice/crio-8d4eeb67f5a50e7de6547207dc98c0c41cfc802622d1c77c8ff49847ae0093bd WatchSource:0}: Error finding container 8d4eeb67f5a50e7de6547207dc98c0c41cfc802622d1c77c8ff49847ae0093bd: Status 404 returned error can't find the container with id 8d4eeb67f5a50e7de6547207dc98c0c41cfc802622d1c77c8ff49847ae0093bd Jul 16 21:51:47.561710 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:47.561446 2577 generic.go:356] "Generic (PLEG): container finished" podID="484e5b37-973b-4ca4-bfa3-399e9e60f196" containerID="7cfabc209b0c724fd17b741f85d984d391cfb9f3dad8bebc3b7c0b7a228bf73b" exitCode=6 Jul 16 21:51:47.561710 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:47.561610 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" event={"ID":"484e5b37-973b-4ca4-bfa3-399e9e60f196","Type":"ContainerDied","Data":"7cfabc209b0c724fd17b741f85d984d391cfb9f3dad8bebc3b7c0b7a228bf73b"} Jul 16 21:51:47.561924 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:47.561822 2577 scope.go:117] "RemoveContainer" containerID="7cfabc209b0c724fd17b741f85d984d391cfb9f3dad8bebc3b7c0b7a228bf73b" Jul 16 21:51:47.563105 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:47.563080 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6" event={"ID":"20d8967d-45cb-455f-b2b6-26d8034e70af","Type":"ContainerStarted","Data":"8d4eeb67f5a50e7de6547207dc98c0c41cfc802622d1c77c8ff49847ae0093bd"} Jul 16 21:51:48.569750 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:48.569663 2577 generic.go:356] "Generic (PLEG): container finished" podID="484e5b37-973b-4ca4-bfa3-399e9e60f196" containerID="dbc0dcee26f81c92d6653a460f9aaacee86578311ec507ca60b102a4cba2b863" exitCode=6 Jul 16 21:51:48.569750 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:48.569729 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" event={"ID":"484e5b37-973b-4ca4-bfa3-399e9e60f196","Type":"ContainerDied","Data":"dbc0dcee26f81c92d6653a460f9aaacee86578311ec507ca60b102a4cba2b863"} Jul 16 21:51:48.570379 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:48.569789 2577 scope.go:117] "RemoveContainer" containerID="7cfabc209b0c724fd17b741f85d984d391cfb9f3dad8bebc3b7c0b7a228bf73b" Jul 16 21:51:48.570379 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:48.569961 2577 scope.go:117] "RemoveContainer" containerID="dbc0dcee26f81c92d6653a460f9aaacee86578311ec507ca60b102a4cba2b863" Jul 16 21:51:48.570379 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:48.570223 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g_odh-ai-gateway-infra(484e5b37-973b-4ca4-bfa3-399e9e60f196)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" podUID="484e5b37-973b-4ca4-bfa3-399e9e60f196" Jul 16 21:51:48.571395 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:48.571373 2577 generic.go:356] "Generic (PLEG): container finished" podID="20d8967d-45cb-455f-b2b6-26d8034e70af" containerID="f760f246d139488a451ad43bd2fdc05782f1e54a4c9c3b36e5f8860108dccb84" exitCode=6 Jul 16 21:51:48.571500 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:48.571417 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6" event={"ID":"20d8967d-45cb-455f-b2b6-26d8034e70af","Type":"ContainerDied","Data":"f760f246d139488a451ad43bd2fdc05782f1e54a4c9c3b36e5f8860108dccb84"} Jul 16 21:51:48.571607 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:48.571593 2577 scope.go:117] "RemoveContainer" containerID="f760f246d139488a451ad43bd2fdc05782f1e54a4c9c3b36e5f8860108dccb84" Jul 16 21:51:49.577866 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:49.577777 2577 generic.go:356] "Generic (PLEG): container finished" podID="20d8967d-45cb-455f-b2b6-26d8034e70af" containerID="45508e01c9cff0617b4162ae1121ee06907ad9572964cc4c5b47d3eed36b4cb5" exitCode=6 Jul 16 21:51:49.577866 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:49.577852 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6" event={"ID":"20d8967d-45cb-455f-b2b6-26d8034e70af","Type":"ContainerDied","Data":"45508e01c9cff0617b4162ae1121ee06907ad9572964cc4c5b47d3eed36b4cb5"} Jul 16 21:51:49.578344 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:49.577909 2577 scope.go:117] "RemoveContainer" containerID="f760f246d139488a451ad43bd2fdc05782f1e54a4c9c3b36e5f8860108dccb84" Jul 16 21:51:49.578344 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:49.578072 2577 scope.go:117] "RemoveContainer" containerID="45508e01c9cff0617b4162ae1121ee06907ad9572964cc4c5b47d3eed36b4cb5" Jul 16 21:51:49.578448 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:49.578335 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6_odh-ai-gateway-infra(20d8967d-45cb-455f-b2b6-26d8034e70af)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6" podUID="20d8967d-45cb-455f-b2b6-26d8034e70af" Jul 16 21:51:49.579626 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:49.579608 2577 scope.go:117] "RemoveContainer" containerID="dbc0dcee26f81c92d6653a460f9aaacee86578311ec507ca60b102a4cba2b863" Jul 16 21:51:49.579763 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:49.579748 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g_odh-ai-gateway-infra(484e5b37-973b-4ca4-bfa3-399e9e60f196)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" podUID="484e5b37-973b-4ca4-bfa3-399e9e60f196" Jul 16 21:51:50.586224 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:50.586197 2577 scope.go:117] "RemoveContainer" containerID="45508e01c9cff0617b4162ae1121ee06907ad9572964cc4c5b47d3eed36b4cb5" Jul 16 21:51:50.586588 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:50.586365 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6_odh-ai-gateway-infra(20d8967d-45cb-455f-b2b6-26d8034e70af)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6" podUID="20d8967d-45cb-455f-b2b6-26d8034e70af" Jul 16 21:51:53.371303 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:53.371267 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx"] Jul 16 21:51:53.374267 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:53.374252 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx" Jul 16 21:51:53.383815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:53.383793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c40734db-1132-4f04-9290-ebca06463f60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx\" (UID: \"c40734db-1132-4f04-9290-ebca06463f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx" Jul 16 21:51:53.395459 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:53.395435 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx"] Jul 16 21:51:53.485173 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:53.485122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c40734db-1132-4f04-9290-ebca06463f60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx\" (UID: \"c40734db-1132-4f04-9290-ebca06463f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx" Jul 16 21:51:53.485732 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:53.485713 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c40734db-1132-4f04-9290-ebca06463f60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx\" (UID: \"c40734db-1132-4f04-9290-ebca06463f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx" Jul 16 21:51:53.683712 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:53.683667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx" Jul 16 21:51:53.812747 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:53.812723 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx"] Jul 16 21:51:53.814275 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:51:53.814253 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40734db_1132_4f04_9290_ebca06463f60.slice/crio-e49eb09f9ea92fe67945b5105a014395ae953f5d00a57fe976f917882ba73b87 WatchSource:0}: Error finding container e49eb09f9ea92fe67945b5105a014395ae953f5d00a57fe976f917882ba73b87: Status 404 returned error can't find the container with id e49eb09f9ea92fe67945b5105a014395ae953f5d00a57fe976f917882ba73b87 Jul 16 21:51:54.609650 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:54.609607 2577 generic.go:356] "Generic (PLEG): container finished" podID="c40734db-1132-4f04-9290-ebca06463f60" containerID="258bec748f757c62446cbeee9043573edc66e375c3d256892c5c7a4bf165bc13" exitCode=6 Jul 16 21:51:54.610040 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:54.609663 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx" event={"ID":"c40734db-1132-4f04-9290-ebca06463f60","Type":"ContainerDied","Data":"258bec748f757c62446cbeee9043573edc66e375c3d256892c5c7a4bf165bc13"} Jul 16 21:51:54.610040 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:54.609711 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx" event={"ID":"c40734db-1132-4f04-9290-ebca06463f60","Type":"ContainerStarted","Data":"e49eb09f9ea92fe67945b5105a014395ae953f5d00a57fe976f917882ba73b87"} Jul 16 21:51:54.610040 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:54.609942 2577 scope.go:117] "RemoveContainer" containerID="258bec748f757c62446cbeee9043573edc66e375c3d256892c5c7a4bf165bc13" Jul 16 21:51:55.116592 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:55.116499 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q"] Jul 16 21:51:55.119833 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:55.119812 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q" Jul 16 21:51:55.129265 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:55.129238 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q"] Jul 16 21:51:55.199247 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:55.199213 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48cbf250-90dd-4040-941b-8fd003b58c9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q\" (UID: \"48cbf250-90dd-4040-941b-8fd003b58c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q" Jul 16 21:51:55.300387 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:55.300358 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48cbf250-90dd-4040-941b-8fd003b58c9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q\" (UID: \"48cbf250-90dd-4040-941b-8fd003b58c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q" Jul 16 21:51:55.300924 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:55.300906 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48cbf250-90dd-4040-941b-8fd003b58c9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q\" (UID: \"48cbf250-90dd-4040-941b-8fd003b58c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q" Jul 16 21:51:55.431278 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:55.431244 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q" Jul 16 21:51:55.563907 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:55.563880 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q"] Jul 16 21:51:55.565984 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:51:55.565957 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cbf250_90dd_4040_941b_8fd003b58c9f.slice/crio-fba710048b2a7c7d446687ca33498c96b1dc6b91f8b67fab8ea9df403e0c1f76 WatchSource:0}: Error finding container fba710048b2a7c7d446687ca33498c96b1dc6b91f8b67fab8ea9df403e0c1f76: Status 404 returned error can't find the container with id fba710048b2a7c7d446687ca33498c96b1dc6b91f8b67fab8ea9df403e0c1f76 Jul 16 21:51:55.616274 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:55.616250 2577 generic.go:356] "Generic (PLEG): container finished" podID="c40734db-1132-4f04-9290-ebca06463f60" containerID="6319e73028e1b939dfc2c163cf1cbf31a0d6738860e6d76ac16618976aa2dd3b" exitCode=6 Jul 16 21:51:55.616637 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:55.616318 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx" event={"ID":"c40734db-1132-4f04-9290-ebca06463f60","Type":"ContainerDied","Data":"6319e73028e1b939dfc2c163cf1cbf31a0d6738860e6d76ac16618976aa2dd3b"} Jul 16 21:51:55.616637 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:55.616362 2577 scope.go:117] "RemoveContainer" containerID="258bec748f757c62446cbeee9043573edc66e375c3d256892c5c7a4bf165bc13" Jul 16 21:51:55.616637 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:55.616573 2577 scope.go:117] "RemoveContainer" containerID="6319e73028e1b939dfc2c163cf1cbf31a0d6738860e6d76ac16618976aa2dd3b" Jul 16 21:51:55.616806 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:55.616784 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx_odh-ai-gateway-infra(c40734db-1132-4f04-9290-ebca06463f60)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx" podUID="c40734db-1132-4f04-9290-ebca06463f60" Jul 16 21:51:55.617620 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:55.617600 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q" event={"ID":"48cbf250-90dd-4040-941b-8fd003b58c9f","Type":"ContainerStarted","Data":"fba710048b2a7c7d446687ca33498c96b1dc6b91f8b67fab8ea9df403e0c1f76"} Jul 16 21:51:56.625086 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:56.624991 2577 scope.go:117] "RemoveContainer" containerID="6319e73028e1b939dfc2c163cf1cbf31a0d6738860e6d76ac16618976aa2dd3b" Jul 16 21:51:56.625517 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:56.625243 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx_odh-ai-gateway-infra(c40734db-1132-4f04-9290-ebca06463f60)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx" podUID="c40734db-1132-4f04-9290-ebca06463f60" Jul 16 21:51:56.626257 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:56.626230 2577 generic.go:356] "Generic (PLEG): container finished" podID="48cbf250-90dd-4040-941b-8fd003b58c9f" containerID="5bbc2d346965bf19114ff127f89a1a7e4de7026e312fd7cc5ed4a4579ed3bb27" exitCode=6 Jul 16 21:51:56.626338 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:56.626313 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q" event={"ID":"48cbf250-90dd-4040-941b-8fd003b58c9f","Type":"ContainerDied","Data":"5bbc2d346965bf19114ff127f89a1a7e4de7026e312fd7cc5ed4a4579ed3bb27"} Jul 16 21:51:56.626433 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:56.626419 2577 scope.go:117] "RemoveContainer" containerID="5bbc2d346965bf19114ff127f89a1a7e4de7026e312fd7cc5ed4a4579ed3bb27" Jul 16 21:51:57.633577 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:57.633539 2577 generic.go:356] "Generic (PLEG): container finished" podID="48cbf250-90dd-4040-941b-8fd003b58c9f" containerID="e241ee8192eeb398e1482ffce241c97476bbf2518c600c454685a0eba0441bd3" exitCode=6 Jul 16 21:51:57.634046 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:57.633622 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q" event={"ID":"48cbf250-90dd-4040-941b-8fd003b58c9f","Type":"ContainerDied","Data":"e241ee8192eeb398e1482ffce241c97476bbf2518c600c454685a0eba0441bd3"} Jul 16 21:51:57.634046 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:57.633698 2577 scope.go:117] "RemoveContainer" containerID="5bbc2d346965bf19114ff127f89a1a7e4de7026e312fd7cc5ed4a4579ed3bb27" Jul 16 21:51:57.634046 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:57.633869 2577 scope.go:117] "RemoveContainer" containerID="e241ee8192eeb398e1482ffce241c97476bbf2518c600c454685a0eba0441bd3" Jul 16 21:51:57.634160 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:57.634058 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q_odh-ai-gateway-infra(48cbf250-90dd-4040-941b-8fd003b58c9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q" podUID="48cbf250-90dd-4040-941b-8fd003b58c9f" Jul 16 21:51:58.640162 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:51:58.640132 2577 scope.go:117] "RemoveContainer" containerID="e241ee8192eeb398e1482ffce241c97476bbf2518c600c454685a0eba0441bd3" Jul 16 21:51:58.640581 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:51:58.640344 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q_odh-ai-gateway-infra(48cbf250-90dd-4040-941b-8fd003b58c9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q" podUID="48cbf250-90dd-4040-941b-8fd003b58c9f" Jul 16 21:52:01.445477 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:01.445446 2577 scope.go:117] "RemoveContainer" containerID="dbc0dcee26f81c92d6653a460f9aaacee86578311ec507ca60b102a4cba2b863" Jul 16 21:52:02.662554 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:02.662519 2577 generic.go:356] "Generic (PLEG): container finished" podID="484e5b37-973b-4ca4-bfa3-399e9e60f196" containerID="f8cf50fa4e2906d95cefd580856f64de0e3328b728d41b2941d1c6388b57b009" exitCode=6 Jul 16 21:52:02.662957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:02.662608 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" event={"ID":"484e5b37-973b-4ca4-bfa3-399e9e60f196","Type":"ContainerDied","Data":"f8cf50fa4e2906d95cefd580856f64de0e3328b728d41b2941d1c6388b57b009"} Jul 16 21:52:02.662957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:02.662674 2577 scope.go:117] "RemoveContainer" containerID="dbc0dcee26f81c92d6653a460f9aaacee86578311ec507ca60b102a4cba2b863" Jul 16 21:52:02.663067 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:02.662967 2577 scope.go:117] "RemoveContainer" containerID="f8cf50fa4e2906d95cefd580856f64de0e3328b728d41b2941d1c6388b57b009" Jul 16 21:52:02.663224 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:52:02.663206 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g_odh-ai-gateway-infra(484e5b37-973b-4ca4-bfa3-399e9e60f196)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" podUID="484e5b37-973b-4ca4-bfa3-399e9e60f196" Jul 16 21:52:03.445035 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:03.445007 2577 scope.go:117] "RemoveContainer" containerID="45508e01c9cff0617b4162ae1121ee06907ad9572964cc4c5b47d3eed36b4cb5" Jul 16 21:52:03.698782 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:03.698720 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g"] Jul 16 21:52:03.825951 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:03.825931 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" Jul 16 21:52:03.877704 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:03.877677 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/484e5b37-973b-4ca4-bfa3-399e9e60f196-maas-api-service-ca\") pod \"484e5b37-973b-4ca4-bfa3-399e9e60f196\" (UID: \"484e5b37-973b-4ca4-bfa3-399e9e60f196\") " Jul 16 21:52:03.877997 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:03.877977 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/484e5b37-973b-4ca4-bfa3-399e9e60f196-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "484e5b37-973b-4ca4-bfa3-399e9e60f196" (UID: "484e5b37-973b-4ca4-bfa3-399e9e60f196"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:52:03.979139 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:03.979064 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/484e5b37-973b-4ca4-bfa3-399e9e60f196-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:52:04.675559 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:04.675529 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" Jul 16 21:52:04.675776 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:04.675562 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g" event={"ID":"484e5b37-973b-4ca4-bfa3-399e9e60f196","Type":"ContainerDied","Data":"ecec805b9de53f78140037a8fad7c21860127ef66cfa2ea4667eab90837bded2"} Jul 16 21:52:04.675776 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:04.675628 2577 scope.go:117] "RemoveContainer" containerID="f8cf50fa4e2906d95cefd580856f64de0e3328b728d41b2941d1c6388b57b009" Jul 16 21:52:04.677841 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:04.677732 2577 generic.go:356] "Generic (PLEG): container finished" podID="20d8967d-45cb-455f-b2b6-26d8034e70af" containerID="5609ce38cf7b2e06b9642c91413715be804ba5395ca3ac73f4e7d70b7ab7e77b" exitCode=6 Jul 16 21:52:04.677841 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:04.677791 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6" event={"ID":"20d8967d-45cb-455f-b2b6-26d8034e70af","Type":"ContainerDied","Data":"5609ce38cf7b2e06b9642c91413715be804ba5395ca3ac73f4e7d70b7ab7e77b"} Jul 16 21:52:04.678114 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:04.678096 2577 scope.go:117] "RemoveContainer" containerID="5609ce38cf7b2e06b9642c91413715be804ba5395ca3ac73f4e7d70b7ab7e77b" Jul 16 21:52:04.678397 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:52:04.678374 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6_odh-ai-gateway-infra(20d8967d-45cb-455f-b2b6-26d8034e70af)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6" podUID="20d8967d-45cb-455f-b2b6-26d8034e70af" Jul 16 21:52:04.684605 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:04.684590 2577 scope.go:117] "RemoveContainer" containerID="45508e01c9cff0617b4162ae1121ee06907ad9572964cc4c5b47d3eed36b4cb5" Jul 16 21:52:04.694229 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:04.694206 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g"] Jul 16 21:52:04.702398 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:04.702378 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-z2z6g"] Jul 16 21:52:05.731785 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:05.731754 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6"] Jul 16 21:52:05.858750 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:05.858728 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6" Jul 16 21:52:05.894667 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:05.894645 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d8967d-45cb-455f-b2b6-26d8034e70af-maas-api-service-ca\") pod \"20d8967d-45cb-455f-b2b6-26d8034e70af\" (UID: \"20d8967d-45cb-455f-b2b6-26d8034e70af\") " Jul 16 21:52:05.894981 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:05.894961 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d8967d-45cb-455f-b2b6-26d8034e70af-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "20d8967d-45cb-455f-b2b6-26d8034e70af" (UID: "20d8967d-45cb-455f-b2b6-26d8034e70af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:52:05.995595 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:05.995534 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d8967d-45cb-455f-b2b6-26d8034e70af-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:52:06.450872 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:06.450839 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="484e5b37-973b-4ca4-bfa3-399e9e60f196" path="/var/lib/kubelet/pods/484e5b37-973b-4ca4-bfa3-399e9e60f196/volumes" Jul 16 21:52:06.691678 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:06.691648 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6" event={"ID":"20d8967d-45cb-455f-b2b6-26d8034e70af","Type":"ContainerDied","Data":"8d4eeb67f5a50e7de6547207dc98c0c41cfc802622d1c77c8ff49847ae0093bd"} Jul 16 21:52:06.691840 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:06.691701 2577 scope.go:117] "RemoveContainer" containerID="5609ce38cf7b2e06b9642c91413715be804ba5395ca3ac73f4e7d70b7ab7e77b" Jul 16 21:52:06.691840 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:06.691661 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6" Jul 16 21:52:06.710649 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:06.710601 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6"] Jul 16 21:52:06.714504 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:06.714485 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-n5gw6"] Jul 16 21:52:08.451249 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:08.451214 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20d8967d-45cb-455f-b2b6-26d8034e70af" path="/var/lib/kubelet/pods/20d8967d-45cb-455f-b2b6-26d8034e70af/volumes" Jul 16 21:52:09.445595 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:09.445568 2577 scope.go:117] "RemoveContainer" containerID="6319e73028e1b939dfc2c163cf1cbf31a0d6738860e6d76ac16618976aa2dd3b" Jul 16 21:52:10.444979 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:10.444950 2577 scope.go:117] "RemoveContainer" containerID="e241ee8192eeb398e1482ffce241c97476bbf2518c600c454685a0eba0441bd3" Jul 16 21:52:10.719289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:10.719261 2577 generic.go:356] "Generic (PLEG): container finished" podID="c40734db-1132-4f04-9290-ebca06463f60" containerID="8a21ddecd72f5b9dcaeeb317e91cb7d12421927119154162f7ab1bc34a39f987" exitCode=6 Jul 16 21:52:10.719420 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:10.719302 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx" event={"ID":"c40734db-1132-4f04-9290-ebca06463f60","Type":"ContainerDied","Data":"8a21ddecd72f5b9dcaeeb317e91cb7d12421927119154162f7ab1bc34a39f987"} Jul 16 21:52:10.719420 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:10.719350 2577 scope.go:117] "RemoveContainer" containerID="6319e73028e1b939dfc2c163cf1cbf31a0d6738860e6d76ac16618976aa2dd3b" Jul 16 21:52:10.719680 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:10.719661 2577 scope.go:117] "RemoveContainer" containerID="8a21ddecd72f5b9dcaeeb317e91cb7d12421927119154162f7ab1bc34a39f987" Jul 16 21:52:10.719883 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:52:10.719861 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx_odh-ai-gateway-infra(c40734db-1132-4f04-9290-ebca06463f60)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx" podUID="c40734db-1132-4f04-9290-ebca06463f60" Jul 16 21:52:11.727479 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:11.727443 2577 generic.go:356] "Generic (PLEG): container finished" podID="48cbf250-90dd-4040-941b-8fd003b58c9f" containerID="4ae2f9a26fcd27dd48b5255089e103068fc387bf481b4712384793b4843c4d97" exitCode=6 Jul 16 21:52:11.727885 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:11.727513 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q" event={"ID":"48cbf250-90dd-4040-941b-8fd003b58c9f","Type":"ContainerDied","Data":"4ae2f9a26fcd27dd48b5255089e103068fc387bf481b4712384793b4843c4d97"} Jul 16 21:52:11.727885 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:11.727571 2577 scope.go:117] "RemoveContainer" containerID="e241ee8192eeb398e1482ffce241c97476bbf2518c600c454685a0eba0441bd3" Jul 16 21:52:11.727885 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:11.727840 2577 scope.go:117] "RemoveContainer" containerID="4ae2f9a26fcd27dd48b5255089e103068fc387bf481b4712384793b4843c4d97" Jul 16 21:52:11.728027 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:52:11.728006 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q_odh-ai-gateway-infra(48cbf250-90dd-4040-941b-8fd003b58c9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q" podUID="48cbf250-90dd-4040-941b-8fd003b58c9f" Jul 16 21:52:11.757916 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:11.757892 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx"] Jul 16 21:52:11.888942 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:11.888921 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx" Jul 16 21:52:11.952328 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:11.951368 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c40734db-1132-4f04-9290-ebca06463f60-maas-api-service-ca\") pod \"c40734db-1132-4f04-9290-ebca06463f60\" (UID: \"c40734db-1132-4f04-9290-ebca06463f60\") " Jul 16 21:52:11.952328 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:11.952062 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40734db-1132-4f04-9290-ebca06463f60-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c40734db-1132-4f04-9290-ebca06463f60" (UID: "c40734db-1132-4f04-9290-ebca06463f60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:52:12.053092 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:12.053027 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c40734db-1132-4f04-9290-ebca06463f60-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:52:12.734272 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:12.734196 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx" Jul 16 21:52:12.734272 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:12.734201 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx" event={"ID":"c40734db-1132-4f04-9290-ebca06463f60","Type":"ContainerDied","Data":"e49eb09f9ea92fe67945b5105a014395ae953f5d00a57fe976f917882ba73b87"} Jul 16 21:52:12.734272 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:12.734269 2577 scope.go:117] "RemoveContainer" containerID="8a21ddecd72f5b9dcaeeb317e91cb7d12421927119154162f7ab1bc34a39f987" Jul 16 21:52:12.761409 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:12.761372 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx"] Jul 16 21:52:12.763455 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:12.763433 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q"] Jul 16 21:52:12.768991 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:12.768971 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-qw5gx"] Jul 16 21:52:12.911720 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:12.911698 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q" Jul 16 21:52:12.961503 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:12.961477 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48cbf250-90dd-4040-941b-8fd003b58c9f-maas-api-service-ca\") pod \"48cbf250-90dd-4040-941b-8fd003b58c9f\" (UID: \"48cbf250-90dd-4040-941b-8fd003b58c9f\") " Jul 16 21:52:12.961801 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:12.961781 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48cbf250-90dd-4040-941b-8fd003b58c9f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48cbf250-90dd-4040-941b-8fd003b58c9f" (UID: "48cbf250-90dd-4040-941b-8fd003b58c9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:52:13.062607 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:13.062522 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48cbf250-90dd-4040-941b-8fd003b58c9f-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:52:13.743068 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:13.743039 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q" Jul 16 21:52:13.743539 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:13.743087 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q" event={"ID":"48cbf250-90dd-4040-941b-8fd003b58c9f","Type":"ContainerDied","Data":"fba710048b2a7c7d446687ca33498c96b1dc6b91f8b67fab8ea9df403e0c1f76"} Jul 16 21:52:13.743539 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:13.743140 2577 scope.go:117] "RemoveContainer" containerID="4ae2f9a26fcd27dd48b5255089e103068fc387bf481b4712384793b4843c4d97" Jul 16 21:52:13.766652 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:13.766620 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q"] Jul 16 21:52:13.772137 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:13.772112 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-n5z7q"] Jul 16 21:52:14.451028 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:14.450997 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48cbf250-90dd-4040-941b-8fd003b58c9f" path="/var/lib/kubelet/pods/48cbf250-90dd-4040-941b-8fd003b58c9f/volumes" Jul 16 21:52:14.451378 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:14.451363 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c40734db-1132-4f04-9290-ebca06463f60" path="/var/lib/kubelet/pods/c40734db-1132-4f04-9290-ebca06463f60/volumes" Jul 16 21:52:35.757819 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.757785 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m"] Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758126 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20d8967d-45cb-455f-b2b6-26d8034e70af" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758138 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="20d8967d-45cb-455f-b2b6-26d8034e70af" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758146 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40734db-1132-4f04-9290-ebca06463f60" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758152 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40734db-1132-4f04-9290-ebca06463f60" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758166 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20d8967d-45cb-455f-b2b6-26d8034e70af" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758171 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="20d8967d-45cb-455f-b2b6-26d8034e70af" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758178 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40734db-1132-4f04-9290-ebca06463f60" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758203 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40734db-1132-4f04-9290-ebca06463f60" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758212 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40734db-1132-4f04-9290-ebca06463f60" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758217 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40734db-1132-4f04-9290-ebca06463f60" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758224 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484e5b37-973b-4ca4-bfa3-399e9e60f196" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758229 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="484e5b37-973b-4ca4-bfa3-399e9e60f196" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758234 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48cbf250-90dd-4040-941b-8fd003b58c9f" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758239 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="48cbf250-90dd-4040-941b-8fd003b58c9f" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758248 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48cbf250-90dd-4040-941b-8fd003b58c9f" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758252 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="48cbf250-90dd-4040-941b-8fd003b58c9f" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758260 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484e5b37-973b-4ca4-bfa3-399e9e60f196" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758265 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="484e5b37-973b-4ca4-bfa3-399e9e60f196" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758272 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484e5b37-973b-4ca4-bfa3-399e9e60f196" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758279 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="484e5b37-973b-4ca4-bfa3-399e9e60f196" containerName="revoke-keys" Jul 16 21:52:35.758277 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758288 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20d8967d-45cb-455f-b2b6-26d8034e70af" containerName="revoke-keys" Jul 16 21:52:35.758914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758294 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="20d8967d-45cb-455f-b2b6-26d8034e70af" containerName="revoke-keys" Jul 16 21:52:35.758914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758345 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40734db-1132-4f04-9290-ebca06463f60" containerName="revoke-keys" Jul 16 21:52:35.758914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758355 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="484e5b37-973b-4ca4-bfa3-399e9e60f196" containerName="revoke-keys" Jul 16 21:52:35.758914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758365 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="48cbf250-90dd-4040-941b-8fd003b58c9f" containerName="revoke-keys" Jul 16 21:52:35.758914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758370 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="484e5b37-973b-4ca4-bfa3-399e9e60f196" containerName="revoke-keys" Jul 16 21:52:35.758914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758376 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="484e5b37-973b-4ca4-bfa3-399e9e60f196" containerName="revoke-keys" Jul 16 21:52:35.758914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758381 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="48cbf250-90dd-4040-941b-8fd003b58c9f" containerName="revoke-keys" Jul 16 21:52:35.758914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758386 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="20d8967d-45cb-455f-b2b6-26d8034e70af" containerName="revoke-keys" Jul 16 21:52:35.758914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758391 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="20d8967d-45cb-455f-b2b6-26d8034e70af" containerName="revoke-keys" Jul 16 21:52:35.758914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758396 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40734db-1132-4f04-9290-ebca06463f60" containerName="revoke-keys" Jul 16 21:52:35.758914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758401 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="48cbf250-90dd-4040-941b-8fd003b58c9f" containerName="revoke-keys" Jul 16 21:52:35.758914 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.758406 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="20d8967d-45cb-455f-b2b6-26d8034e70af" containerName="revoke-keys" Jul 16 21:52:35.761192 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.761166 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" Jul 16 21:52:35.764190 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.764168 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:52:35.764320 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.764225 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:52:35.769753 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.769727 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m"] Jul 16 21:52:35.858606 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.858582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e87833-c212-4ad4-8f11-a66b5c54984b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m\" (UID: \"78e87833-c212-4ad4-8f11-a66b5c54984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" Jul 16 21:52:35.959768 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.959738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e87833-c212-4ad4-8f11-a66b5c54984b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m\" (UID: \"78e87833-c212-4ad4-8f11-a66b5c54984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" Jul 16 21:52:35.960352 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:35.960336 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e87833-c212-4ad4-8f11-a66b5c54984b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m\" (UID: \"78e87833-c212-4ad4-8f11-a66b5c54984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" Jul 16 21:52:36.072487 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:36.072428 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" Jul 16 21:52:36.202749 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:36.202728 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m"] Jul 16 21:52:36.204690 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:52:36.204665 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e87833_c212_4ad4_8f11_a66b5c54984b.slice/crio-0782474975a423c46e904dd188852747535f9c41919447d4b1109cc7e8185804 WatchSource:0}: Error finding container 0782474975a423c46e904dd188852747535f9c41919447d4b1109cc7e8185804: Status 404 returned error can't find the container with id 0782474975a423c46e904dd188852747535f9c41919447d4b1109cc7e8185804 Jul 16 21:52:36.860652 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:36.860621 2577 generic.go:356] "Generic (PLEG): container finished" podID="78e87833-c212-4ad4-8f11-a66b5c54984b" containerID="b5fd84855fb9e43ac622f3f67f92d296560cd12cb9e9a8ea7bef4771beaf329b" exitCode=6 Jul 16 21:52:36.860976 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:36.860659 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" event={"ID":"78e87833-c212-4ad4-8f11-a66b5c54984b","Type":"ContainerDied","Data":"b5fd84855fb9e43ac622f3f67f92d296560cd12cb9e9a8ea7bef4771beaf329b"} Jul 16 21:52:36.860976 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:36.860714 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" event={"ID":"78e87833-c212-4ad4-8f11-a66b5c54984b","Type":"ContainerStarted","Data":"0782474975a423c46e904dd188852747535f9c41919447d4b1109cc7e8185804"} Jul 16 21:52:36.860976 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:36.860899 2577 scope.go:117] "RemoveContainer" containerID="b5fd84855fb9e43ac622f3f67f92d296560cd12cb9e9a8ea7bef4771beaf329b" Jul 16 21:52:36.922020 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:36.921996 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p"] Jul 16 21:52:36.922363 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:36.922351 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48cbf250-90dd-4040-941b-8fd003b58c9f" containerName="revoke-keys" Jul 16 21:52:36.922413 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:36.922365 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="48cbf250-90dd-4040-941b-8fd003b58c9f" containerName="revoke-keys" Jul 16 21:52:36.922449 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:36.922433 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40734db-1132-4f04-9290-ebca06463f60" containerName="revoke-keys" Jul 16 21:52:36.926561 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:36.926547 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" Jul 16 21:52:36.935395 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:36.935371 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p"] Jul 16 21:52:37.071460 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:37.071393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3678279-904d-4174-bc4c-3d20aabd04fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p\" (UID: \"d3678279-904d-4174-bc4c-3d20aabd04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" Jul 16 21:52:37.172616 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:37.172590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3678279-904d-4174-bc4c-3d20aabd04fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p\" (UID: \"d3678279-904d-4174-bc4c-3d20aabd04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" Jul 16 21:52:37.173143 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:37.173125 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3678279-904d-4174-bc4c-3d20aabd04fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p\" (UID: \"d3678279-904d-4174-bc4c-3d20aabd04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" Jul 16 21:52:37.237062 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:37.237040 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" Jul 16 21:52:37.364701 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:37.364672 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p"] Jul 16 21:52:37.366701 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:52:37.366675 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3678279_904d_4174_bc4c_3d20aabd04fb.slice/crio-2e7307e9c493c2a69a89c6d8f49d2d8869083a0277344b8f4c8d269e04e699ce WatchSource:0}: Error finding container 2e7307e9c493c2a69a89c6d8f49d2d8869083a0277344b8f4c8d269e04e699ce: Status 404 returned error can't find the container with id 2e7307e9c493c2a69a89c6d8f49d2d8869083a0277344b8f4c8d269e04e699ce Jul 16 21:52:37.866915 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:37.866883 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" event={"ID":"d3678279-904d-4174-bc4c-3d20aabd04fb","Type":"ContainerStarted","Data":"fdf5f3ee29eb8c73989291fcb8c868f51affaf3552791b2715d610c8056ba79c"} Jul 16 21:52:37.867346 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:37.866945 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" event={"ID":"d3678279-904d-4174-bc4c-3d20aabd04fb","Type":"ContainerStarted","Data":"2e7307e9c493c2a69a89c6d8f49d2d8869083a0277344b8f4c8d269e04e699ce"} Jul 16 21:52:37.868754 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:37.868734 2577 generic.go:356] "Generic (PLEG): container finished" podID="78e87833-c212-4ad4-8f11-a66b5c54984b" containerID="93060a48a2f54f2b00685fa82eb587c313a5ad2fab635877e41e711f72ce7669" exitCode=6 Jul 16 21:52:37.868819 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:37.868784 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" event={"ID":"78e87833-c212-4ad4-8f11-a66b5c54984b","Type":"ContainerDied","Data":"93060a48a2f54f2b00685fa82eb587c313a5ad2fab635877e41e711f72ce7669"} Jul 16 21:52:37.868861 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:37.868824 2577 scope.go:117] "RemoveContainer" containerID="b5fd84855fb9e43ac622f3f67f92d296560cd12cb9e9a8ea7bef4771beaf329b" Jul 16 21:52:37.868956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:37.868930 2577 scope.go:117] "RemoveContainer" containerID="93060a48a2f54f2b00685fa82eb587c313a5ad2fab635877e41e711f72ce7669" Jul 16 21:52:37.869163 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:52:37.869140 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m_odh-ai-gateway-infra(78e87833-c212-4ad4-8f11-a66b5c54984b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" podUID="78e87833-c212-4ad4-8f11-a66b5c54984b" Jul 16 21:52:37.883643 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:37.883600 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" podStartSLOduration=1.8835860819999999 podStartE2EDuration="1.883586082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:52:37.882377601 +0000 UTC m=+2574.057485814" watchObservedRunningTime="2026-07-16 21:52:37.883586082 +0000 UTC m=+2574.058694274" Jul 16 21:52:38.875653 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:38.875618 2577 generic.go:356] "Generic (PLEG): container finished" podID="d3678279-904d-4174-bc4c-3d20aabd04fb" containerID="fdf5f3ee29eb8c73989291fcb8c868f51affaf3552791b2715d610c8056ba79c" exitCode=6 Jul 16 21:52:38.876127 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:38.875702 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" event={"ID":"d3678279-904d-4174-bc4c-3d20aabd04fb","Type":"ContainerDied","Data":"fdf5f3ee29eb8c73989291fcb8c868f51affaf3552791b2715d610c8056ba79c"} Jul 16 21:52:38.876127 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:38.875954 2577 scope.go:117] "RemoveContainer" containerID="fdf5f3ee29eb8c73989291fcb8c868f51affaf3552791b2715d610c8056ba79c" Jul 16 21:52:38.877448 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:38.877429 2577 scope.go:117] "RemoveContainer" containerID="93060a48a2f54f2b00685fa82eb587c313a5ad2fab635877e41e711f72ce7669" Jul 16 21:52:38.877602 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:52:38.877586 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m_odh-ai-gateway-infra(78e87833-c212-4ad4-8f11-a66b5c54984b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" podUID="78e87833-c212-4ad4-8f11-a66b5c54984b" Jul 16 21:52:39.884394 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:39.884358 2577 generic.go:356] "Generic (PLEG): container finished" podID="d3678279-904d-4174-bc4c-3d20aabd04fb" containerID="d2bc86c59de574aaa27ad01cdfd0c003ec8a1cde00c4d3c43c547962d4873e09" exitCode=6 Jul 16 21:52:39.884790 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:39.884444 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" event={"ID":"d3678279-904d-4174-bc4c-3d20aabd04fb","Type":"ContainerDied","Data":"d2bc86c59de574aaa27ad01cdfd0c003ec8a1cde00c4d3c43c547962d4873e09"} Jul 16 21:52:39.884790 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:39.884508 2577 scope.go:117] "RemoveContainer" containerID="fdf5f3ee29eb8c73989291fcb8c868f51affaf3552791b2715d610c8056ba79c" Jul 16 21:52:39.884790 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:39.884712 2577 scope.go:117] "RemoveContainer" containerID="d2bc86c59de574aaa27ad01cdfd0c003ec8a1cde00c4d3c43c547962d4873e09" Jul 16 21:52:39.884963 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:52:39.884939 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p_odh-ai-gateway-infra(d3678279-904d-4174-bc4c-3d20aabd04fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" podUID="d3678279-904d-4174-bc4c-3d20aabd04fb" Jul 16 21:52:40.891314 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:40.891289 2577 scope.go:117] "RemoveContainer" containerID="d2bc86c59de574aaa27ad01cdfd0c003ec8a1cde00c4d3c43c547962d4873e09" Jul 16 21:52:40.891680 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:52:40.891455 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p_odh-ai-gateway-infra(d3678279-904d-4174-bc4c-3d20aabd04fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" podUID="d3678279-904d-4174-bc4c-3d20aabd04fb" Jul 16 21:52:43.404560 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:43.404530 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz"] Jul 16 21:52:43.409308 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:43.409289 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" Jul 16 21:52:43.417301 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:43.417276 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz"] Jul 16 21:52:43.422173 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:43.422150 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7be174-e556-456a-a601-d9a8804b55cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz\" (UID: \"5e7be174-e556-456a-a601-d9a8804b55cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" Jul 16 21:52:43.523225 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:43.523177 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7be174-e556-456a-a601-d9a8804b55cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz\" (UID: \"5e7be174-e556-456a-a601-d9a8804b55cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" Jul 16 21:52:43.523780 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:43.523760 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7be174-e556-456a-a601-d9a8804b55cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz\" (UID: \"5e7be174-e556-456a-a601-d9a8804b55cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" Jul 16 21:52:43.721080 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:43.720995 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" Jul 16 21:52:43.852590 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:43.852568 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz"] Jul 16 21:52:43.853505 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:52:43.853483 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7be174_e556_456a_a601_d9a8804b55cd.slice/crio-188c8aad1f6a7cf870f7524a6f82765bbbcd9455e7a20a9b9586849cdd314b6c WatchSource:0}: Error finding container 188c8aad1f6a7cf870f7524a6f82765bbbcd9455e7a20a9b9586849cdd314b6c: Status 404 returned error can't find the container with id 188c8aad1f6a7cf870f7524a6f82765bbbcd9455e7a20a9b9586849cdd314b6c Jul 16 21:52:43.908330 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:43.908306 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" event={"ID":"5e7be174-e556-456a-a601-d9a8804b55cd","Type":"ContainerStarted","Data":"188c8aad1f6a7cf870f7524a6f82765bbbcd9455e7a20a9b9586849cdd314b6c"} Jul 16 21:52:44.914236 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:44.914202 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e7be174-e556-456a-a601-d9a8804b55cd" containerID="ec2dfda58362bce6414276e9e4e87d6fb0216c38afae8131917d272d9c80683f" exitCode=6 Jul 16 21:52:44.914623 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:44.914292 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" event={"ID":"5e7be174-e556-456a-a601-d9a8804b55cd","Type":"ContainerDied","Data":"ec2dfda58362bce6414276e9e4e87d6fb0216c38afae8131917d272d9c80683f"} Jul 16 21:52:44.914623 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:44.914452 2577 scope.go:117] "RemoveContainer" containerID="ec2dfda58362bce6414276e9e4e87d6fb0216c38afae8131917d272d9c80683f" Jul 16 21:52:45.921291 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:45.921263 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e7be174-e556-456a-a601-d9a8804b55cd" containerID="9b11dba50aa7f41e05744285b19b32de548080786b824e447e7044bdd701124b" exitCode=6 Jul 16 21:52:45.921689 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:45.921296 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" event={"ID":"5e7be174-e556-456a-a601-d9a8804b55cd","Type":"ContainerDied","Data":"9b11dba50aa7f41e05744285b19b32de548080786b824e447e7044bdd701124b"} Jul 16 21:52:45.921689 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:45.921342 2577 scope.go:117] "RemoveContainer" containerID="ec2dfda58362bce6414276e9e4e87d6fb0216c38afae8131917d272d9c80683f" Jul 16 21:52:45.921689 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:45.921510 2577 scope.go:117] "RemoveContainer" containerID="9b11dba50aa7f41e05744285b19b32de548080786b824e447e7044bdd701124b" Jul 16 21:52:45.921811 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:52:45.921741 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz_odh-ai-gateway-infra(5e7be174-e556-456a-a601-d9a8804b55cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" podUID="5e7be174-e556-456a-a601-d9a8804b55cd" Jul 16 21:52:46.928671 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:46.928649 2577 scope.go:117] "RemoveContainer" containerID="9b11dba50aa7f41e05744285b19b32de548080786b824e447e7044bdd701124b" Jul 16 21:52:46.929037 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:52:46.928818 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz_odh-ai-gateway-infra(5e7be174-e556-456a-a601-d9a8804b55cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" podUID="5e7be174-e556-456a-a601-d9a8804b55cd" Jul 16 21:52:52.444676 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:52.444649 2577 scope.go:117] "RemoveContainer" containerID="d2bc86c59de574aaa27ad01cdfd0c003ec8a1cde00c4d3c43c547962d4873e09" Jul 16 21:52:52.960650 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:52.960615 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" event={"ID":"d3678279-904d-4174-bc4c-3d20aabd04fb","Type":"ContainerStarted","Data":"b841f55f709df19b7f775b19b3c0b1ad3a141e05dce66030e87b36ad5f72fa1a"} Jul 16 21:52:53.445359 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:53.445331 2577 scope.go:117] "RemoveContainer" containerID="93060a48a2f54f2b00685fa82eb587c313a5ad2fab635877e41e711f72ce7669" Jul 16 21:52:53.967911 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:53.967884 2577 generic.go:356] "Generic (PLEG): container finished" podID="d3678279-904d-4174-bc4c-3d20aabd04fb" containerID="b841f55f709df19b7f775b19b3c0b1ad3a141e05dce66030e87b36ad5f72fa1a" exitCode=6 Jul 16 21:52:53.968093 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:53.967964 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" event={"ID":"d3678279-904d-4174-bc4c-3d20aabd04fb","Type":"ContainerDied","Data":"b841f55f709df19b7f775b19b3c0b1ad3a141e05dce66030e87b36ad5f72fa1a"} Jul 16 21:52:53.968093 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:53.968031 2577 scope.go:117] "RemoveContainer" containerID="d2bc86c59de574aaa27ad01cdfd0c003ec8a1cde00c4d3c43c547962d4873e09" Jul 16 21:52:53.968348 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:53.968331 2577 scope.go:117] "RemoveContainer" containerID="b841f55f709df19b7f775b19b3c0b1ad3a141e05dce66030e87b36ad5f72fa1a" Jul 16 21:52:53.968555 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:52:53.968531 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p_odh-ai-gateway-infra(d3678279-904d-4174-bc4c-3d20aabd04fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" podUID="d3678279-904d-4174-bc4c-3d20aabd04fb" Jul 16 21:52:53.969815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:53.969797 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" event={"ID":"78e87833-c212-4ad4-8f11-a66b5c54984b","Type":"ContainerStarted","Data":"184f9a2a819c7f327b1255edabe03c80088c5e61502b4622ccbfe3be72d0b39e"} Jul 16 21:52:54.001362 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:54.001340 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p"] Jul 16 21:52:54.009071 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:54.009032 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" podStartSLOduration=19.009019682 podStartE2EDuration="19.009019682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:52:54.007775416 +0000 UTC m=+2590.182883608" watchObservedRunningTime="2026-07-16 21:52:54.009019682 +0000 UTC m=+2590.184127876" Jul 16 21:52:54.978624 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:54.978586 2577 generic.go:356] "Generic (PLEG): container finished" podID="78e87833-c212-4ad4-8f11-a66b5c54984b" containerID="184f9a2a819c7f327b1255edabe03c80088c5e61502b4622ccbfe3be72d0b39e" exitCode=6 Jul 16 21:52:54.978947 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:54.978646 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" event={"ID":"78e87833-c212-4ad4-8f11-a66b5c54984b","Type":"ContainerDied","Data":"184f9a2a819c7f327b1255edabe03c80088c5e61502b4622ccbfe3be72d0b39e"} Jul 16 21:52:54.978947 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:54.978712 2577 scope.go:117] "RemoveContainer" containerID="93060a48a2f54f2b00685fa82eb587c313a5ad2fab635877e41e711f72ce7669" Jul 16 21:52:54.978947 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:54.978922 2577 scope.go:117] "RemoveContainer" containerID="184f9a2a819c7f327b1255edabe03c80088c5e61502b4622ccbfe3be72d0b39e" Jul 16 21:52:54.979165 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:52:54.979146 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m_odh-ai-gateway-infra(78e87833-c212-4ad4-8f11-a66b5c54984b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" podUID="78e87833-c212-4ad4-8f11-a66b5c54984b" Jul 16 21:52:55.024749 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:55.024718 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m"] Jul 16 21:52:55.110531 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:55.110510 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" Jul 16 21:52:55.221589 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:55.221563 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3678279-904d-4174-bc4c-3d20aabd04fb-maas-api-service-ca\") pod \"d3678279-904d-4174-bc4c-3d20aabd04fb\" (UID: \"d3678279-904d-4174-bc4c-3d20aabd04fb\") " Jul 16 21:52:55.221881 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:55.221861 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3678279-904d-4174-bc4c-3d20aabd04fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3678279-904d-4174-bc4c-3d20aabd04fb" (UID: "d3678279-904d-4174-bc4c-3d20aabd04fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:52:55.322801 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:55.322745 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3678279-904d-4174-bc4c-3d20aabd04fb-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:52:55.987871 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:55.987839 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" Jul 16 21:52:55.987871 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:55.987856 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p" event={"ID":"d3678279-904d-4174-bc4c-3d20aabd04fb","Type":"ContainerDied","Data":"2e7307e9c493c2a69a89c6d8f49d2d8869083a0277344b8f4c8d269e04e699ce"} Jul 16 21:52:55.988409 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:55.987919 2577 scope.go:117] "RemoveContainer" containerID="b841f55f709df19b7f775b19b3c0b1ad3a141e05dce66030e87b36ad5f72fa1a" Jul 16 21:52:56.023265 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:56.023233 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p"] Jul 16 21:52:56.027965 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:56.027937 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-2dm6p"] Jul 16 21:52:56.127393 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:56.127368 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" Jul 16 21:52:56.229557 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:56.229534 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e87833-c212-4ad4-8f11-a66b5c54984b-maas-api-service-ca\") pod \"78e87833-c212-4ad4-8f11-a66b5c54984b\" (UID: \"78e87833-c212-4ad4-8f11-a66b5c54984b\") " Jul 16 21:52:56.229890 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:56.229867 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e87833-c212-4ad4-8f11-a66b5c54984b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78e87833-c212-4ad4-8f11-a66b5c54984b" (UID: "78e87833-c212-4ad4-8f11-a66b5c54984b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:52:56.330978 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:56.330918 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e87833-c212-4ad4-8f11-a66b5c54984b-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:52:56.450174 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:56.450148 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3678279-904d-4174-bc4c-3d20aabd04fb" path="/var/lib/kubelet/pods/d3678279-904d-4174-bc4c-3d20aabd04fb/volumes" Jul 16 21:52:56.996079 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:56.996045 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" event={"ID":"78e87833-c212-4ad4-8f11-a66b5c54984b","Type":"ContainerDied","Data":"0782474975a423c46e904dd188852747535f9c41919447d4b1109cc7e8185804"} Jul 16 21:52:56.996545 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:56.996062 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m" Jul 16 21:52:56.996545 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:56.996098 2577 scope.go:117] "RemoveContainer" containerID="184f9a2a819c7f327b1255edabe03c80088c5e61502b4622ccbfe3be72d0b39e" Jul 16 21:52:57.016444 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:57.016423 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m"] Jul 16 21:52:57.020658 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:57.020635 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-klc9m"] Jul 16 21:52:58.450231 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:58.450179 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78e87833-c212-4ad4-8f11-a66b5c54984b" path="/var/lib/kubelet/pods/78e87833-c212-4ad4-8f11-a66b5c54984b/volumes" Jul 16 21:52:59.445370 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:52:59.445342 2577 scope.go:117] "RemoveContainer" containerID="9b11dba50aa7f41e05744285b19b32de548080786b824e447e7044bdd701124b" Jul 16 21:53:00.013699 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:00.013631 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" event={"ID":"5e7be174-e556-456a-a601-d9a8804b55cd","Type":"ContainerStarted","Data":"e836fc902ba515c94230016bd32c0f809f599100f933ceab2d62a4f710b096d8"} Jul 16 21:53:00.033310 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:00.033266 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" podStartSLOduration=17.03325251 podStartE2EDuration="17.03325251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:00.032305981 +0000 UTC m=+2596.207414175" watchObservedRunningTime="2026-07-16 21:53:00.03325251 +0000 UTC m=+2596.208360745" Jul 16 21:53:01.020644 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:01.020612 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e7be174-e556-456a-a601-d9a8804b55cd" containerID="e836fc902ba515c94230016bd32c0f809f599100f933ceab2d62a4f710b096d8" exitCode=6 Jul 16 21:53:01.021054 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:01.020684 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" event={"ID":"5e7be174-e556-456a-a601-d9a8804b55cd","Type":"ContainerDied","Data":"e836fc902ba515c94230016bd32c0f809f599100f933ceab2d62a4f710b096d8"} Jul 16 21:53:01.021054 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:01.020738 2577 scope.go:117] "RemoveContainer" containerID="9b11dba50aa7f41e05744285b19b32de548080786b824e447e7044bdd701124b" Jul 16 21:53:01.021054 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:01.020982 2577 scope.go:117] "RemoveContainer" containerID="e836fc902ba515c94230016bd32c0f809f599100f933ceab2d62a4f710b096d8" Jul 16 21:53:01.021245 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:53:01.021146 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz_odh-ai-gateway-infra(5e7be174-e556-456a-a601-d9a8804b55cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" podUID="5e7be174-e556-456a-a601-d9a8804b55cd" Jul 16 21:53:01.055386 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:01.055355 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz"] Jul 16 21:53:02.155492 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:02.155470 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" Jul 16 21:53:02.275946 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:02.275916 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7be174-e556-456a-a601-d9a8804b55cd-maas-api-service-ca\") pod \"5e7be174-e556-456a-a601-d9a8804b55cd\" (UID: \"5e7be174-e556-456a-a601-d9a8804b55cd\") " Jul 16 21:53:02.276310 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:02.276292 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e7be174-e556-456a-a601-d9a8804b55cd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e7be174-e556-456a-a601-d9a8804b55cd" (UID: "5e7be174-e556-456a-a601-d9a8804b55cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:53:02.377475 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:02.377449 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7be174-e556-456a-a601-d9a8804b55cd-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:53:03.033621 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:03.033589 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" Jul 16 21:53:03.033621 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:03.033601 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz" event={"ID":"5e7be174-e556-456a-a601-d9a8804b55cd","Type":"ContainerDied","Data":"188c8aad1f6a7cf870f7524a6f82765bbbcd9455e7a20a9b9586849cdd314b6c"} Jul 16 21:53:03.033843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:03.033663 2577 scope.go:117] "RemoveContainer" containerID="e836fc902ba515c94230016bd32c0f809f599100f933ceab2d62a4f710b096d8" Jul 16 21:53:03.056500 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:03.054712 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz"] Jul 16 21:53:03.063561 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:03.063537 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-kvwpz"] Jul 16 21:53:04.450649 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:04.450618 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e7be174-e556-456a-a601-d9a8804b55cd" path="/var/lib/kubelet/pods/5e7be174-e556-456a-a601-d9a8804b55cd/volumes" Jul 16 21:53:20.935949 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.935867 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg"] Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936269 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3678279-904d-4174-bc4c-3d20aabd04fb" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936283 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3678279-904d-4174-bc4c-3d20aabd04fb" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936292 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e87833-c212-4ad4-8f11-a66b5c54984b" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936297 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e87833-c212-4ad4-8f11-a66b5c54984b" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936308 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e87833-c212-4ad4-8f11-a66b5c54984b" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936314 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e87833-c212-4ad4-8f11-a66b5c54984b" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936321 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e7be174-e556-456a-a601-d9a8804b55cd" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936329 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e7be174-e556-456a-a601-d9a8804b55cd" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936337 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e7be174-e556-456a-a601-d9a8804b55cd" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936342 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e7be174-e556-456a-a601-d9a8804b55cd" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936354 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3678279-904d-4174-bc4c-3d20aabd04fb" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936360 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3678279-904d-4174-bc4c-3d20aabd04fb" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936418 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3678279-904d-4174-bc4c-3d20aabd04fb" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936427 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3678279-904d-4174-bc4c-3d20aabd04fb" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936435 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e87833-c212-4ad4-8f11-a66b5c54984b" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936441 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e7be174-e556-456a-a601-d9a8804b55cd" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936445 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e7be174-e556-456a-a601-d9a8804b55cd" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936453 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e87833-c212-4ad4-8f11-a66b5c54984b" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936459 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e87833-c212-4ad4-8f11-a66b5c54984b" containerName="revoke-keys" Jul 16 21:53:20.936453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.936463 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e7be174-e556-456a-a601-d9a8804b55cd" containerName="revoke-keys" Jul 16 21:53:20.940694 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.940675 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" Jul 16 21:53:20.945094 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.945073 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xcg5c\"" Jul 16 21:53:20.945229 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.945149 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f8eae57b-serving-cert\"" Jul 16 21:53:20.945290 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.945233 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:53:20.946280 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.946261 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 21:53:20.952762 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:20.952739 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg"] Jul 16 21:53:21.030271 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:21.030238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e281d9-17ce-4271-8b78-7a93e15604bc-openshift-service-ca\") pod \"maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg\" (UID: \"83e281d9-17ce-4271-8b78-7a93e15604bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" Jul 16 21:53:21.030408 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:21.030280 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e281d9-17ce-4271-8b78-7a93e15604bc-maas-api-tls\") pod \"maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg\" (UID: \"83e281d9-17ce-4271-8b78-7a93e15604bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" Jul 16 21:53:21.030451 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:21.030409 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtcbd\" (UniqueName: \"kubernetes.io/projected/83e281d9-17ce-4271-8b78-7a93e15604bc-kube-api-access-wtcbd\") pod \"maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg\" (UID: \"83e281d9-17ce-4271-8b78-7a93e15604bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" Jul 16 21:53:21.131253 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:21.131224 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtcbd\" (UniqueName: \"kubernetes.io/projected/83e281d9-17ce-4271-8b78-7a93e15604bc-kube-api-access-wtcbd\") pod \"maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg\" (UID: \"83e281d9-17ce-4271-8b78-7a93e15604bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" Jul 16 21:53:21.131420 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:21.131259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e281d9-17ce-4271-8b78-7a93e15604bc-openshift-service-ca\") pod \"maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg\" (UID: \"83e281d9-17ce-4271-8b78-7a93e15604bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" Jul 16 21:53:21.131420 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:21.131291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e281d9-17ce-4271-8b78-7a93e15604bc-maas-api-tls\") pod \"maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg\" (UID: \"83e281d9-17ce-4271-8b78-7a93e15604bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" Jul 16 21:53:21.131509 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:53:21.131440 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-serving-cert: secret "maas-api-e2e-mt-f8eae57b-serving-cert" not found Jul 16 21:53:21.131555 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:53:21.131521 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83e281d9-17ce-4271-8b78-7a93e15604bc-maas-api-tls podName:83e281d9-17ce-4271-8b78-7a93e15604bc nodeName:}" failed. No retries permitted until 2026-07-16 21:53:21.631497035 +0000 UTC m=+2617.806605209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83e281d9-17ce-4271-8b78-7a93e15604bc-maas-api-tls") pod "maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" (UID: "83e281d9-17ce-4271-8b78-7a93e15604bc") : secret "maas-api-e2e-mt-f8eae57b-serving-cert" not found Jul 16 21:53:21.131940 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:21.131923 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e281d9-17ce-4271-8b78-7a93e15604bc-openshift-service-ca\") pod \"maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg\" (UID: \"83e281d9-17ce-4271-8b78-7a93e15604bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" Jul 16 21:53:21.145168 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:21.145145 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtcbd\" (UniqueName: \"kubernetes.io/projected/83e281d9-17ce-4271-8b78-7a93e15604bc-kube-api-access-wtcbd\") pod \"maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg\" (UID: \"83e281d9-17ce-4271-8b78-7a93e15604bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" Jul 16 21:53:21.635421 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:21.635381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e281d9-17ce-4271-8b78-7a93e15604bc-maas-api-tls\") pod \"maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg\" (UID: \"83e281d9-17ce-4271-8b78-7a93e15604bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" Jul 16 21:53:21.637875 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:21.637858 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e281d9-17ce-4271-8b78-7a93e15604bc-maas-api-tls\") pod \"maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg\" (UID: \"83e281d9-17ce-4271-8b78-7a93e15604bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" Jul 16 21:53:21.851420 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:21.851385 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" Jul 16 21:53:21.992441 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:21.992416 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg"] Jul 16 21:53:21.994242 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:53:21.994170 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e281d9_17ce_4271_8b78_7a93e15604bc.slice/crio-5f2537a374bc5357eb247f5be3ff01128d89e7dd62b8c24901c42837b208cc4b WatchSource:0}: Error finding container 5f2537a374bc5357eb247f5be3ff01128d89e7dd62b8c24901c42837b208cc4b: Status 404 returned error can't find the container with id 5f2537a374bc5357eb247f5be3ff01128d89e7dd62b8c24901c42837b208cc4b Jul 16 21:53:21.995868 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:21.995848 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:53:22.131721 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:22.131691 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" event={"ID":"83e281d9-17ce-4271-8b78-7a93e15604bc","Type":"ContainerStarted","Data":"5f2537a374bc5357eb247f5be3ff01128d89e7dd62b8c24901c42837b208cc4b"} Jul 16 21:53:23.138761 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:23.138729 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" event={"ID":"83e281d9-17ce-4271-8b78-7a93e15604bc","Type":"ContainerStarted","Data":"97b490fff8672e86496002cdee327816266c557ba5b813f62e82f0fe225e20c7"} Jul 16 21:53:23.139163 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:23.138830 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" Jul 16 21:53:23.159508 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:23.159463 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" podStartSLOduration=2.470114239 podStartE2EDuration="3.159446974s" totalImagesPullingTime="689.332735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:23.157982813 +0000 UTC m=+2619.333091031" watchObservedRunningTime="2026-07-16 21:53:23.159446974 +0000 UTC m=+2619.334555165" Jul 16 21:53:26.983740 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:26.983710 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2"] Jul 16 21:53:26.984202 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:26.984044 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e87833-c212-4ad4-8f11-a66b5c54984b" containerName="revoke-keys" Jul 16 21:53:26.984202 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:26.984055 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e87833-c212-4ad4-8f11-a66b5c54984b" containerName="revoke-keys" Jul 16 21:53:26.984202 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:26.984062 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e7be174-e556-456a-a601-d9a8804b55cd" containerName="revoke-keys" Jul 16 21:53:26.984202 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:26.984068 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e7be174-e556-456a-a601-d9a8804b55cd" containerName="revoke-keys" Jul 16 21:53:26.984202 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:26.984082 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3678279-904d-4174-bc4c-3d20aabd04fb" containerName="revoke-keys" Jul 16 21:53:26.984202 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:26.984087 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3678279-904d-4174-bc4c-3d20aabd04fb" containerName="revoke-keys" Jul 16 21:53:26.984202 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:26.984147 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3678279-904d-4174-bc4c-3d20aabd04fb" containerName="revoke-keys" Jul 16 21:53:26.987162 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:26.987147 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2" Jul 16 21:53:26.990810 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:26.990783 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:53:26.998696 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:26.998672 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2"] Jul 16 21:53:27.088790 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:27.088765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2\" (UID: \"ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2" Jul 16 21:53:27.189639 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:27.189613 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2\" (UID: \"ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2" Jul 16 21:53:27.190173 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:27.190155 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2\" (UID: \"ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2" Jul 16 21:53:27.297461 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:27.297394 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2" Jul 16 21:53:27.426967 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:27.426944 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2"] Jul 16 21:53:27.429326 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:53:27.429297 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce72c2e6_108c_4aa8_bfb8_5a68ccfbc9d4.slice/crio-b47424ea8bdf5e386d493203a4b457aa6ae599d5235bd3834dda0535aa38fb78 WatchSource:0}: Error finding container b47424ea8bdf5e386d493203a4b457aa6ae599d5235bd3834dda0535aa38fb78: Status 404 returned error can't find the container with id b47424ea8bdf5e386d493203a4b457aa6ae599d5235bd3834dda0535aa38fb78 Jul 16 21:53:28.164741 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:28.164708 2577 generic.go:356] "Generic (PLEG): container finished" podID="ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" containerID="94210f11aaf9af561d6bd17721edac5340a34d73dbc5502a8477687115b52dc1" exitCode=6 Jul 16 21:53:28.165154 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:28.164789 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2" event={"ID":"ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4","Type":"ContainerDied","Data":"94210f11aaf9af561d6bd17721edac5340a34d73dbc5502a8477687115b52dc1"} Jul 16 21:53:28.165154 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:28.164840 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2" event={"ID":"ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4","Type":"ContainerStarted","Data":"b47424ea8bdf5e386d493203a4b457aa6ae599d5235bd3834dda0535aa38fb78"} Jul 16 21:53:28.165154 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:28.165009 2577 scope.go:117] "RemoveContainer" containerID="94210f11aaf9af561d6bd17721edac5340a34d73dbc5502a8477687115b52dc1" Jul 16 21:53:29.151110 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:29.151086 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" Jul 16 21:53:29.172483 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:29.172450 2577 generic.go:356] "Generic (PLEG): container finished" podID="ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" containerID="219f44db7b7e7800dc84593639952f47265a51e17f3cb66a4b25dacc3679072e" exitCode=6 Jul 16 21:53:29.172892 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:29.172512 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2" event={"ID":"ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4","Type":"ContainerDied","Data":"219f44db7b7e7800dc84593639952f47265a51e17f3cb66a4b25dacc3679072e"} Jul 16 21:53:29.172892 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:29.172563 2577 scope.go:117] "RemoveContainer" containerID="94210f11aaf9af561d6bd17721edac5340a34d73dbc5502a8477687115b52dc1" Jul 16 21:53:29.172892 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:29.172812 2577 scope.go:117] "RemoveContainer" containerID="219f44db7b7e7800dc84593639952f47265a51e17f3cb66a4b25dacc3679072e" Jul 16 21:53:29.173021 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:53:29.173003 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2_odh-ai-gateway-infra(ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2" podUID="ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" Jul 16 21:53:30.185764 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:30.185737 2577 scope.go:117] "RemoveContainer" containerID="219f44db7b7e7800dc84593639952f47265a51e17f3cb66a4b25dacc3679072e" Jul 16 21:53:30.186153 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:53:30.185927 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2_odh-ai-gateway-infra(ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2" podUID="ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" Jul 16 21:53:32.792452 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:32.792419 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr"] Jul 16 21:53:32.797094 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:32.797075 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr" Jul 16 21:53:32.804637 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:32.804610 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr"] Jul 16 21:53:32.835509 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:32.835480 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4059e66f-188c-4ae4-a47b-dc575fe02494-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr\" (UID: \"4059e66f-188c-4ae4-a47b-dc575fe02494\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr" Jul 16 21:53:32.935998 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:32.935971 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4059e66f-188c-4ae4-a47b-dc575fe02494-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr\" (UID: \"4059e66f-188c-4ae4-a47b-dc575fe02494\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr" Jul 16 21:53:32.936574 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:32.936554 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4059e66f-188c-4ae4-a47b-dc575fe02494-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr\" (UID: \"4059e66f-188c-4ae4-a47b-dc575fe02494\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr" Jul 16 21:53:33.108351 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:33.108254 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr" Jul 16 21:53:33.242289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:33.242259 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr"] Jul 16 21:53:33.243667 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:53:33.243640 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4059e66f_188c_4ae4_a47b_dc575fe02494.slice/crio-6c5ecf4fddb3d0c29c536933d12f5e990b26dcaed510566c58b673564bfa3bd7 WatchSource:0}: Error finding container 6c5ecf4fddb3d0c29c536933d12f5e990b26dcaed510566c58b673564bfa3bd7: Status 404 returned error can't find the container with id 6c5ecf4fddb3d0c29c536933d12f5e990b26dcaed510566c58b673564bfa3bd7 Jul 16 21:53:33.457053 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:33.457020 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf"] Jul 16 21:53:33.461543 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:33.461528 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" Jul 16 21:53:33.472217 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:33.472194 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf"] Jul 16 21:53:33.540046 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:33.540021 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11aa1311-51e2-42a4-a6a2-dc2571a456f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf\" (UID: \"11aa1311-51e2-42a4-a6a2-dc2571a456f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" Jul 16 21:53:33.641149 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:33.641113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11aa1311-51e2-42a4-a6a2-dc2571a456f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf\" (UID: \"11aa1311-51e2-42a4-a6a2-dc2571a456f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" Jul 16 21:53:33.641855 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:33.641836 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11aa1311-51e2-42a4-a6a2-dc2571a456f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf\" (UID: \"11aa1311-51e2-42a4-a6a2-dc2571a456f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" Jul 16 21:53:33.772614 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:33.772523 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" Jul 16 21:53:33.903961 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:33.902503 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf"] Jul 16 21:53:34.208325 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:34.208289 2577 generic.go:356] "Generic (PLEG): container finished" podID="4059e66f-188c-4ae4-a47b-dc575fe02494" containerID="4b0307d39436adb56e6338a28feb8b90407083a9c4668019f75105f10332e99d" exitCode=0 Jul 16 21:53:34.208505 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:34.208374 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr" event={"ID":"4059e66f-188c-4ae4-a47b-dc575fe02494","Type":"ContainerDied","Data":"4b0307d39436adb56e6338a28feb8b90407083a9c4668019f75105f10332e99d"} Jul 16 21:53:34.208505 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:34.208424 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr" event={"ID":"4059e66f-188c-4ae4-a47b-dc575fe02494","Type":"ContainerStarted","Data":"6c5ecf4fddb3d0c29c536933d12f5e990b26dcaed510566c58b673564bfa3bd7"} Jul 16 21:53:34.209823 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:34.209795 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" event={"ID":"11aa1311-51e2-42a4-a6a2-dc2571a456f5","Type":"ContainerStarted","Data":"6427cded9a38fc4f9ec627ca4e40f1eda9695143ad5e5fd89307b4408974b63f"} Jul 16 21:53:34.209934 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:34.209840 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" event={"ID":"11aa1311-51e2-42a4-a6a2-dc2571a456f5","Type":"ContainerStarted","Data":"50da09f8b3b1602ed3831b0fdcfc7e7326f677df3c0926c50b9bbc8e5a70d9d2"} Jul 16 21:53:34.242367 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:34.242328 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" podStartSLOduration=1.242313321 podStartE2EDuration="1.242313321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:34.241270468 +0000 UTC m=+2630.416378660" watchObservedRunningTime="2026-07-16 21:53:34.242313321 +0000 UTC m=+2630.417421512" Jul 16 21:53:35.215906 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:35.215874 2577 generic.go:356] "Generic (PLEG): container finished" podID="11aa1311-51e2-42a4-a6a2-dc2571a456f5" containerID="6427cded9a38fc4f9ec627ca4e40f1eda9695143ad5e5fd89307b4408974b63f" exitCode=6 Jul 16 21:53:35.216380 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:35.215945 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" event={"ID":"11aa1311-51e2-42a4-a6a2-dc2571a456f5","Type":"ContainerDied","Data":"6427cded9a38fc4f9ec627ca4e40f1eda9695143ad5e5fd89307b4408974b63f"} Jul 16 21:53:35.216380 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:35.216116 2577 scope.go:117] "RemoveContainer" containerID="6427cded9a38fc4f9ec627ca4e40f1eda9695143ad5e5fd89307b4408974b63f" Jul 16 21:53:35.346785 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:35.346762 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr" Jul 16 21:53:35.353220 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:35.353200 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4059e66f-188c-4ae4-a47b-dc575fe02494-maas-api-service-ca\") pod \"4059e66f-188c-4ae4-a47b-dc575fe02494\" (UID: \"4059e66f-188c-4ae4-a47b-dc575fe02494\") " Jul 16 21:53:35.353523 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:35.353501 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4059e66f-188c-4ae4-a47b-dc575fe02494-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4059e66f-188c-4ae4-a47b-dc575fe02494" (UID: "4059e66f-188c-4ae4-a47b-dc575fe02494"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:53:35.453719 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:35.453687 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4059e66f-188c-4ae4-a47b-dc575fe02494-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:53:35.813017 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:35.812911 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh"] Jul 16 21:53:35.813393 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:35.813375 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4059e66f-188c-4ae4-a47b-dc575fe02494" containerName="revoke-keys" Jul 16 21:53:35.813504 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:35.813395 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4059e66f-188c-4ae4-a47b-dc575fe02494" containerName="revoke-keys" Jul 16 21:53:35.813563 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:35.813521 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4059e66f-188c-4ae4-a47b-dc575fe02494" containerName="revoke-keys" Jul 16 21:53:35.816591 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:35.816567 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh" Jul 16 21:53:35.825322 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:35.825295 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh"] Jul 16 21:53:35.855839 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:35.855809 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c3259b-f997-4ae2-9361-5e48d57700c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh\" (UID: \"f1c3259b-f997-4ae2-9361-5e48d57700c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh" Jul 16 21:53:35.956596 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:35.956562 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c3259b-f997-4ae2-9361-5e48d57700c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh\" (UID: \"f1c3259b-f997-4ae2-9361-5e48d57700c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh" Jul 16 21:53:35.957170 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:35.957151 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c3259b-f997-4ae2-9361-5e48d57700c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh\" (UID: \"f1c3259b-f997-4ae2-9361-5e48d57700c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh" Jul 16 21:53:36.128974 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:36.128946 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh" Jul 16 21:53:36.246586 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:36.246544 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr" event={"ID":"4059e66f-188c-4ae4-a47b-dc575fe02494","Type":"ContainerDied","Data":"6c5ecf4fddb3d0c29c536933d12f5e990b26dcaed510566c58b673564bfa3bd7"} Jul 16 21:53:36.247036 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:36.246614 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c5ecf4fddb3d0c29c536933d12f5e990b26dcaed510566c58b673564bfa3bd7" Jul 16 21:53:36.247036 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:36.246700 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr" Jul 16 21:53:36.249321 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:36.249271 2577 generic.go:356] "Generic (PLEG): container finished" podID="11aa1311-51e2-42a4-a6a2-dc2571a456f5" containerID="6157d523b5c2e46249e114411a8d86d42971ecb9292160a930d64c573a9fbd3c" exitCode=6 Jul 16 21:53:36.249462 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:36.249398 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" event={"ID":"11aa1311-51e2-42a4-a6a2-dc2571a456f5","Type":"ContainerDied","Data":"6157d523b5c2e46249e114411a8d86d42971ecb9292160a930d64c573a9fbd3c"} Jul 16 21:53:36.249533 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:36.249474 2577 scope.go:117] "RemoveContainer" containerID="6427cded9a38fc4f9ec627ca4e40f1eda9695143ad5e5fd89307b4408974b63f" Jul 16 21:53:36.249764 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:36.249643 2577 scope.go:117] "RemoveContainer" containerID="6157d523b5c2e46249e114411a8d86d42971ecb9292160a930d64c573a9fbd3c" Jul 16 21:53:36.250133 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:53:36.249886 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf_odh-ai-gateway-infra(11aa1311-51e2-42a4-a6a2-dc2571a456f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" podUID="11aa1311-51e2-42a4-a6a2-dc2571a456f5" Jul 16 21:53:36.296200 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:36.296020 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh"] Jul 16 21:53:36.298741 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:53:36.298714 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c3259b_f997_4ae2_9361_5e48d57700c0.slice/crio-8d98450b115c27b4e6ba255261a7a7d902c71830a3dc188bf83eac8ee0d4974e WatchSource:0}: Error finding container 8d98450b115c27b4e6ba255261a7a7d902c71830a3dc188bf83eac8ee0d4974e: Status 404 returned error can't find the container with id 8d98450b115c27b4e6ba255261a7a7d902c71830a3dc188bf83eac8ee0d4974e Jul 16 21:53:37.256576 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:37.256491 2577 generic.go:356] "Generic (PLEG): container finished" podID="f1c3259b-f997-4ae2-9361-5e48d57700c0" containerID="ba22bd8e6badaa3f66b084e79b39b74ff2a7baca4f09281d40513e1ea386819b" exitCode=6 Jul 16 21:53:37.256576 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:37.256541 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh" event={"ID":"f1c3259b-f997-4ae2-9361-5e48d57700c0","Type":"ContainerDied","Data":"ba22bd8e6badaa3f66b084e79b39b74ff2a7baca4f09281d40513e1ea386819b"} Jul 16 21:53:37.257017 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:37.256645 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh" event={"ID":"f1c3259b-f997-4ae2-9361-5e48d57700c0","Type":"ContainerStarted","Data":"8d98450b115c27b4e6ba255261a7a7d902c71830a3dc188bf83eac8ee0d4974e"} Jul 16 21:53:37.257017 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:37.256870 2577 scope.go:117] "RemoveContainer" containerID="ba22bd8e6badaa3f66b084e79b39b74ff2a7baca4f09281d40513e1ea386819b" Jul 16 21:53:37.258418 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:37.258400 2577 scope.go:117] "RemoveContainer" containerID="6157d523b5c2e46249e114411a8d86d42971ecb9292160a930d64c573a9fbd3c" Jul 16 21:53:37.258563 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:53:37.258547 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf_odh-ai-gateway-infra(11aa1311-51e2-42a4-a6a2-dc2571a456f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" podUID="11aa1311-51e2-42a4-a6a2-dc2571a456f5" Jul 16 21:53:38.265686 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:38.265607 2577 generic.go:356] "Generic (PLEG): container finished" podID="f1c3259b-f997-4ae2-9361-5e48d57700c0" containerID="8bbb950d7c0d83a7815833233c31a7986adf4d6c45e8750143e6c0143ac27442" exitCode=6 Jul 16 21:53:38.265686 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:38.265649 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh" event={"ID":"f1c3259b-f997-4ae2-9361-5e48d57700c0","Type":"ContainerDied","Data":"8bbb950d7c0d83a7815833233c31a7986adf4d6c45e8750143e6c0143ac27442"} Jul 16 21:53:38.266211 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:38.265698 2577 scope.go:117] "RemoveContainer" containerID="ba22bd8e6badaa3f66b084e79b39b74ff2a7baca4f09281d40513e1ea386819b" Jul 16 21:53:38.266211 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:38.265906 2577 scope.go:117] "RemoveContainer" containerID="8bbb950d7c0d83a7815833233c31a7986adf4d6c45e8750143e6c0143ac27442" Jul 16 21:53:38.266211 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:53:38.266117 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh_odh-ai-gateway-infra(f1c3259b-f997-4ae2-9361-5e48d57700c0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh" podUID="f1c3259b-f997-4ae2-9361-5e48d57700c0" Jul 16 21:53:39.272571 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:39.272543 2577 scope.go:117] "RemoveContainer" containerID="8bbb950d7c0d83a7815833233c31a7986adf4d6c45e8750143e6c0143ac27442" Jul 16 21:53:39.272970 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:53:39.272720 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh_odh-ai-gateway-infra(f1c3259b-f997-4ae2-9361-5e48d57700c0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh" podUID="f1c3259b-f997-4ae2-9361-5e48d57700c0" Jul 16 21:53:43.274567 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:43.274531 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg"] Jul 16 21:53:43.275006 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:43.274784 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" podUID="83e281d9-17ce-4271-8b78-7a93e15604bc" containerName="maas-api" containerID="cri-o://97b490fff8672e86496002cdee327816266c557ba5b813f62e82f0fe225e20c7" gracePeriod=30 Jul 16 21:53:43.525122 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:43.525059 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" Jul 16 21:53:43.611392 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:43.611360 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e281d9-17ce-4271-8b78-7a93e15604bc-maas-api-tls\") pod \"83e281d9-17ce-4271-8b78-7a93e15604bc\" (UID: \"83e281d9-17ce-4271-8b78-7a93e15604bc\") " Jul 16 21:53:43.611552 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:43.611418 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e281d9-17ce-4271-8b78-7a93e15604bc-openshift-service-ca\") pod \"83e281d9-17ce-4271-8b78-7a93e15604bc\" (UID: \"83e281d9-17ce-4271-8b78-7a93e15604bc\") " Jul 16 21:53:43.611552 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:43.611439 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtcbd\" (UniqueName: \"kubernetes.io/projected/83e281d9-17ce-4271-8b78-7a93e15604bc-kube-api-access-wtcbd\") pod \"83e281d9-17ce-4271-8b78-7a93e15604bc\" (UID: \"83e281d9-17ce-4271-8b78-7a93e15604bc\") " Jul 16 21:53:43.611786 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:43.611762 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e281d9-17ce-4271-8b78-7a93e15604bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83e281d9-17ce-4271-8b78-7a93e15604bc" (UID: "83e281d9-17ce-4271-8b78-7a93e15604bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:53:43.613480 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:43.613455 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e281d9-17ce-4271-8b78-7a93e15604bc-kube-api-access-wtcbd" (OuterVolumeSpecName: "kube-api-access-wtcbd") pod "83e281d9-17ce-4271-8b78-7a93e15604bc" (UID: "83e281d9-17ce-4271-8b78-7a93e15604bc"). InnerVolumeSpecName "kube-api-access-wtcbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:53:43.613480 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:43.613463 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83e281d9-17ce-4271-8b78-7a93e15604bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83e281d9-17ce-4271-8b78-7a93e15604bc" (UID: "83e281d9-17ce-4271-8b78-7a93e15604bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:43.712332 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:43.712286 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e281d9-17ce-4271-8b78-7a93e15604bc-maas-api-tls\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:53:43.712332 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:43.712325 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e281d9-17ce-4271-8b78-7a93e15604bc-openshift-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:53:43.712332 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:43.712340 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtcbd\" (UniqueName: \"kubernetes.io/projected/83e281d9-17ce-4271-8b78-7a93e15604bc-kube-api-access-wtcbd\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:53:44.306891 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:44.306858 2577 generic.go:356] "Generic (PLEG): container finished" podID="83e281d9-17ce-4271-8b78-7a93e15604bc" containerID="97b490fff8672e86496002cdee327816266c557ba5b813f62e82f0fe225e20c7" exitCode=0 Jul 16 21:53:44.307309 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:44.306911 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" Jul 16 21:53:44.307309 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:44.306924 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" event={"ID":"83e281d9-17ce-4271-8b78-7a93e15604bc","Type":"ContainerDied","Data":"97b490fff8672e86496002cdee327816266c557ba5b813f62e82f0fe225e20c7"} Jul 16 21:53:44.307309 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:44.306993 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg" event={"ID":"83e281d9-17ce-4271-8b78-7a93e15604bc","Type":"ContainerDied","Data":"5f2537a374bc5357eb247f5be3ff01128d89e7dd62b8c24901c42837b208cc4b"} Jul 16 21:53:44.307309 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:44.307016 2577 scope.go:117] "RemoveContainer" containerID="97b490fff8672e86496002cdee327816266c557ba5b813f62e82f0fe225e20c7" Jul 16 21:53:44.315783 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:44.315768 2577 scope.go:117] "RemoveContainer" containerID="97b490fff8672e86496002cdee327816266c557ba5b813f62e82f0fe225e20c7" Jul 16 21:53:44.316043 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:53:44.316025 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b490fff8672e86496002cdee327816266c557ba5b813f62e82f0fe225e20c7\": container with ID starting with 97b490fff8672e86496002cdee327816266c557ba5b813f62e82f0fe225e20c7 not found: ID does not exist" containerID="97b490fff8672e86496002cdee327816266c557ba5b813f62e82f0fe225e20c7" Jul 16 21:53:44.316121 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:44.316048 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b490fff8672e86496002cdee327816266c557ba5b813f62e82f0fe225e20c7"} err="failed to get container status \"97b490fff8672e86496002cdee327816266c557ba5b813f62e82f0fe225e20c7\": rpc error: code = NotFound desc = could not find container \"97b490fff8672e86496002cdee327816266c557ba5b813f62e82f0fe225e20c7\": container with ID starting with 97b490fff8672e86496002cdee327816266c557ba5b813f62e82f0fe225e20c7 not found: ID does not exist" Jul 16 21:53:44.331682 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:44.331653 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg"] Jul 16 21:53:44.335317 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:44.335295 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8eae57b-6564f588cc-ncpsg"] Jul 16 21:53:44.451390 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:44.451364 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83e281d9-17ce-4271-8b78-7a93e15604bc" path="/var/lib/kubelet/pods/83e281d9-17ce-4271-8b78-7a93e15604bc/volumes" Jul 16 21:53:45.216770 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:45.216728 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs"] Jul 16 21:53:45.217163 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:45.217146 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e281d9-17ce-4271-8b78-7a93e15604bc" containerName="maas-api" Jul 16 21:53:45.217305 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:45.217166 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e281d9-17ce-4271-8b78-7a93e15604bc" containerName="maas-api" Jul 16 21:53:45.217305 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:45.217279 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e281d9-17ce-4271-8b78-7a93e15604bc" containerName="maas-api" Jul 16 21:53:45.221796 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:45.221776 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs" Jul 16 21:53:45.224205 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:45.224175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf21542b-399b-4734-9d44-3622209f237a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-xslrs\" (UID: \"cf21542b-399b-4734-9d44-3622209f237a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs" Jul 16 21:53:45.229987 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:45.229964 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs"] Jul 16 21:53:45.325695 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:45.325663 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf21542b-399b-4734-9d44-3622209f237a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-xslrs\" (UID: \"cf21542b-399b-4734-9d44-3622209f237a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs" Jul 16 21:53:45.326275 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:45.326257 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf21542b-399b-4734-9d44-3622209f237a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-xslrs\" (UID: \"cf21542b-399b-4734-9d44-3622209f237a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs" Jul 16 21:53:45.445020 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:45.444990 2577 scope.go:117] "RemoveContainer" containerID="219f44db7b7e7800dc84593639952f47265a51e17f3cb66a4b25dacc3679072e" Jul 16 21:53:45.534041 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:45.534015 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs" Jul 16 21:53:45.663363 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:45.663333 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs"] Jul 16 21:53:45.664383 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:53:45.664359 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf21542b_399b_4734_9d44_3622209f237a.slice/crio-31a5526623bcbe8bcadca3e546eed2a6011394b8b5e9cc215b40da0be5966cdd WatchSource:0}: Error finding container 31a5526623bcbe8bcadca3e546eed2a6011394b8b5e9cc215b40da0be5966cdd: Status 404 returned error can't find the container with id 31a5526623bcbe8bcadca3e546eed2a6011394b8b5e9cc215b40da0be5966cdd Jul 16 21:53:46.319756 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:46.319730 2577 generic.go:356] "Generic (PLEG): container finished" podID="cf21542b-399b-4734-9d44-3622209f237a" containerID="12d85ada6c4a897a7420c46be849b0a90870972c85de67bf86c72872ada4f891" exitCode=6 Jul 16 21:53:46.319896 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:46.319809 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs" event={"ID":"cf21542b-399b-4734-9d44-3622209f237a","Type":"ContainerDied","Data":"12d85ada6c4a897a7420c46be849b0a90870972c85de67bf86c72872ada4f891"} Jul 16 21:53:46.319896 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:46.319854 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs" event={"ID":"cf21542b-399b-4734-9d44-3622209f237a","Type":"ContainerStarted","Data":"31a5526623bcbe8bcadca3e546eed2a6011394b8b5e9cc215b40da0be5966cdd"} Jul 16 21:53:46.320007 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:46.319962 2577 scope.go:117] "RemoveContainer" containerID="12d85ada6c4a897a7420c46be849b0a90870972c85de67bf86c72872ada4f891" Jul 16 21:53:46.321736 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:46.321717 2577 generic.go:356] "Generic (PLEG): container finished" podID="ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" containerID="17ad19c682397ac30b20daa27a8ee4551a43c55b994c08453971a5bb6b757796" exitCode=6 Jul 16 21:53:46.321844 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:46.321765 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2" event={"ID":"ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4","Type":"ContainerDied","Data":"17ad19c682397ac30b20daa27a8ee4551a43c55b994c08453971a5bb6b757796"} Jul 16 21:53:46.321844 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:46.321802 2577 scope.go:117] "RemoveContainer" containerID="219f44db7b7e7800dc84593639952f47265a51e17f3cb66a4b25dacc3679072e" Jul 16 21:53:46.322132 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:46.322049 2577 scope.go:117] "RemoveContainer" containerID="17ad19c682397ac30b20daa27a8ee4551a43c55b994c08453971a5bb6b757796" Jul 16 21:53:46.322279 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:53:46.322261 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2_odh-ai-gateway-infra(ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2" podUID="ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" Jul 16 21:53:47.328475 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:47.328393 2577 generic.go:356] "Generic (PLEG): container finished" podID="cf21542b-399b-4734-9d44-3622209f237a" containerID="b14992cee11c8a96be4829764df8aa066dd0724ddf8c6a72e225887a179a3a06" exitCode=6 Jul 16 21:53:47.328900 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:47.328476 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs" event={"ID":"cf21542b-399b-4734-9d44-3622209f237a","Type":"ContainerDied","Data":"b14992cee11c8a96be4829764df8aa066dd0724ddf8c6a72e225887a179a3a06"} Jul 16 21:53:47.328900 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:47.328526 2577 scope.go:117] "RemoveContainer" containerID="12d85ada6c4a897a7420c46be849b0a90870972c85de67bf86c72872ada4f891" Jul 16 21:53:47.328900 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:47.328651 2577 scope.go:117] "RemoveContainer" containerID="b14992cee11c8a96be4829764df8aa066dd0724ddf8c6a72e225887a179a3a06" Jul 16 21:53:47.328900 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:53:47.328882 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-xslrs_odh-ai-gateway-infra(cf21542b-399b-4734-9d44-3622209f237a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs" podUID="cf21542b-399b-4734-9d44-3622209f237a" Jul 16 21:53:47.364453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:47.364425 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2"] Jul 16 21:53:47.497660 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:47.497635 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2" Jul 16 21:53:47.642025 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:47.641996 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4-maas-api-service-ca\") pod \"ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4\" (UID: \"ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4\") " Jul 16 21:53:47.642338 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:47.642316 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" (UID: "ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:53:47.743429 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:47.743404 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:53:48.336704 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:48.336675 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2" Jul 16 21:53:48.336704 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:48.336700 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2" event={"ID":"ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4","Type":"ContainerDied","Data":"b47424ea8bdf5e386d493203a4b457aa6ae599d5235bd3834dda0535aa38fb78"} Jul 16 21:53:48.337269 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:48.336789 2577 scope.go:117] "RemoveContainer" containerID="17ad19c682397ac30b20daa27a8ee4551a43c55b994c08453971a5bb6b757796" Jul 16 21:53:48.338567 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:48.338547 2577 scope.go:117] "RemoveContainer" containerID="b14992cee11c8a96be4829764df8aa066dd0724ddf8c6a72e225887a179a3a06" Jul 16 21:53:48.338807 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:53:48.338783 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-xslrs_odh-ai-gateway-infra(cf21542b-399b-4734-9d44-3622209f237a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs" podUID="cf21542b-399b-4734-9d44-3622209f237a" Jul 16 21:53:48.376504 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:48.376480 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2"] Jul 16 21:53:48.382552 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:48.382532 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-vnnl2"] Jul 16 21:53:48.450289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:48.450267 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" path="/var/lib/kubelet/pods/ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4/volumes" Jul 16 21:53:51.444724 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:51.444699 2577 scope.go:117] "RemoveContainer" containerID="8bbb950d7c0d83a7815833233c31a7986adf4d6c45e8750143e6c0143ac27442" Jul 16 21:53:51.445084 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:51.444833 2577 scope.go:117] "RemoveContainer" containerID="6157d523b5c2e46249e114411a8d86d42971ecb9292160a930d64c573a9fbd3c" Jul 16 21:53:52.363099 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:52.363071 2577 generic.go:356] "Generic (PLEG): container finished" podID="11aa1311-51e2-42a4-a6a2-dc2571a456f5" containerID="46cfcdee6f5cfcb0fda8ab4c54777c3197c0ff740ff3b966746f7ec9cf0435eb" exitCode=6 Jul 16 21:53:52.363250 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:52.363144 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" event={"ID":"11aa1311-51e2-42a4-a6a2-dc2571a456f5","Type":"ContainerDied","Data":"46cfcdee6f5cfcb0fda8ab4c54777c3197c0ff740ff3b966746f7ec9cf0435eb"} Jul 16 21:53:52.363250 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:52.363222 2577 scope.go:117] "RemoveContainer" containerID="6157d523b5c2e46249e114411a8d86d42971ecb9292160a930d64c573a9fbd3c" Jul 16 21:53:52.363509 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:52.363493 2577 scope.go:117] "RemoveContainer" containerID="46cfcdee6f5cfcb0fda8ab4c54777c3197c0ff740ff3b966746f7ec9cf0435eb" Jul 16 21:53:52.363765 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:53:52.363727 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf_odh-ai-gateway-infra(11aa1311-51e2-42a4-a6a2-dc2571a456f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" podUID="11aa1311-51e2-42a4-a6a2-dc2571a456f5" Jul 16 21:53:52.365334 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:52.365314 2577 generic.go:356] "Generic (PLEG): container finished" podID="f1c3259b-f997-4ae2-9361-5e48d57700c0" containerID="22cead69ad8b211edca661650a09c0999ef316280861a501cd7015576dab2783" exitCode=6 Jul 16 21:53:52.365424 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:52.365366 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh" event={"ID":"f1c3259b-f997-4ae2-9361-5e48d57700c0","Type":"ContainerDied","Data":"22cead69ad8b211edca661650a09c0999ef316280861a501cd7015576dab2783"} Jul 16 21:53:52.365646 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:52.365632 2577 scope.go:117] "RemoveContainer" containerID="22cead69ad8b211edca661650a09c0999ef316280861a501cd7015576dab2783" Jul 16 21:53:52.365815 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:53:52.365798 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh_odh-ai-gateway-infra(f1c3259b-f997-4ae2-9361-5e48d57700c0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh" podUID="f1c3259b-f997-4ae2-9361-5e48d57700c0" Jul 16 21:53:52.374241 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:52.374225 2577 scope.go:117] "RemoveContainer" containerID="8bbb950d7c0d83a7815833233c31a7986adf4d6c45e8750143e6c0143ac27442" Jul 16 21:53:53.390324 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:53.390289 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf"] Jul 16 21:53:53.405690 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:53.405648 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh"] Jul 16 21:53:53.528947 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:53.528919 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" Jul 16 21:53:53.556011 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:53.555988 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh" Jul 16 21:53:53.583317 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:53.583289 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c3259b-f997-4ae2-9361-5e48d57700c0-maas-api-service-ca\") pod \"f1c3259b-f997-4ae2-9361-5e48d57700c0\" (UID: \"f1c3259b-f997-4ae2-9361-5e48d57700c0\") " Jul 16 21:53:53.583445 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:53.583334 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11aa1311-51e2-42a4-a6a2-dc2571a456f5-maas-api-service-ca\") pod \"11aa1311-51e2-42a4-a6a2-dc2571a456f5\" (UID: \"11aa1311-51e2-42a4-a6a2-dc2571a456f5\") " Jul 16 21:53:53.583651 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:53.583632 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11aa1311-51e2-42a4-a6a2-dc2571a456f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "11aa1311-51e2-42a4-a6a2-dc2571a456f5" (UID: "11aa1311-51e2-42a4-a6a2-dc2571a456f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:53:53.583695 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:53.583642 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c3259b-f997-4ae2-9361-5e48d57700c0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f1c3259b-f997-4ae2-9361-5e48d57700c0" (UID: "f1c3259b-f997-4ae2-9361-5e48d57700c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:53:53.684586 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:53.684513 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11aa1311-51e2-42a4-a6a2-dc2571a456f5-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:53:53.684586 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:53.684538 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c3259b-f997-4ae2-9361-5e48d57700c0-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:53:54.379915 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:54.379883 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh" Jul 16 21:53:54.380123 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:54.379925 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh" event={"ID":"f1c3259b-f997-4ae2-9361-5e48d57700c0","Type":"ContainerDied","Data":"8d98450b115c27b4e6ba255261a7a7d902c71830a3dc188bf83eac8ee0d4974e"} Jul 16 21:53:54.380123 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:54.380002 2577 scope.go:117] "RemoveContainer" containerID="22cead69ad8b211edca661650a09c0999ef316280861a501cd7015576dab2783" Jul 16 21:53:54.381294 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:54.381271 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" event={"ID":"11aa1311-51e2-42a4-a6a2-dc2571a456f5","Type":"ContainerDied","Data":"50da09f8b3b1602ed3831b0fdcfc7e7326f677df3c0926c50b9bbc8e5a70d9d2"} Jul 16 21:53:54.381404 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:54.381311 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf" Jul 16 21:53:54.392292 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:54.392121 2577 scope.go:117] "RemoveContainer" containerID="46cfcdee6f5cfcb0fda8ab4c54777c3197c0ff740ff3b966746f7ec9cf0435eb" Jul 16 21:53:54.410463 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:54.410438 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh"] Jul 16 21:53:54.417541 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:54.417518 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-lw8vh"] Jul 16 21:53:54.429745 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:54.429725 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf"] Jul 16 21:53:54.431629 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:54.431611 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-wz5jf"] Jul 16 21:53:54.450938 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:54.450914 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11aa1311-51e2-42a4-a6a2-dc2571a456f5" path="/var/lib/kubelet/pods/11aa1311-51e2-42a4-a6a2-dc2571a456f5/volumes" Jul 16 21:53:54.451290 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:53:54.451276 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1c3259b-f997-4ae2-9361-5e48d57700c0" path="/var/lib/kubelet/pods/f1c3259b-f997-4ae2-9361-5e48d57700c0/volumes" Jul 16 21:54:02.445222 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:02.445177 2577 scope.go:117] "RemoveContainer" containerID="b14992cee11c8a96be4829764df8aa066dd0724ddf8c6a72e225887a179a3a06" Jul 16 21:54:03.428542 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:03.428512 2577 generic.go:356] "Generic (PLEG): container finished" podID="cf21542b-399b-4734-9d44-3622209f237a" containerID="f8dc8249718b5229199cb5fec54b333bc6134af10aff7d58876551bf6fdaf0e7" exitCode=6 Jul 16 21:54:03.428710 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:03.428546 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs" event={"ID":"cf21542b-399b-4734-9d44-3622209f237a","Type":"ContainerDied","Data":"f8dc8249718b5229199cb5fec54b333bc6134af10aff7d58876551bf6fdaf0e7"} Jul 16 21:54:03.428710 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:03.428592 2577 scope.go:117] "RemoveContainer" containerID="b14992cee11c8a96be4829764df8aa066dd0724ddf8c6a72e225887a179a3a06" Jul 16 21:54:03.428848 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:03.428828 2577 scope.go:117] "RemoveContainer" containerID="f8dc8249718b5229199cb5fec54b333bc6134af10aff7d58876551bf6fdaf0e7" Jul 16 21:54:03.429073 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:54:03.429050 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-xslrs_odh-ai-gateway-infra(cf21542b-399b-4734-9d44-3622209f237a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs" podUID="cf21542b-399b-4734-9d44-3622209f237a" Jul 16 21:54:04.460912 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:04.460880 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs"] Jul 16 21:54:04.591080 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:04.591052 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs" Jul 16 21:54:04.667848 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:04.667820 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf21542b-399b-4734-9d44-3622209f237a-maas-api-service-ca\") pod \"cf21542b-399b-4734-9d44-3622209f237a\" (UID: \"cf21542b-399b-4734-9d44-3622209f237a\") " Jul 16 21:54:04.668158 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:04.668137 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf21542b-399b-4734-9d44-3622209f237a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cf21542b-399b-4734-9d44-3622209f237a" (UID: "cf21542b-399b-4734-9d44-3622209f237a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:54:04.768736 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:04.768662 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf21542b-399b-4734-9d44-3622209f237a-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:54:05.441610 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:05.441579 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs" Jul 16 21:54:05.441801 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:05.441616 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs" event={"ID":"cf21542b-399b-4734-9d44-3622209f237a","Type":"ContainerDied","Data":"31a5526623bcbe8bcadca3e546eed2a6011394b8b5e9cc215b40da0be5966cdd"} Jul 16 21:54:05.441801 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:05.441704 2577 scope.go:117] "RemoveContainer" containerID="f8dc8249718b5229199cb5fec54b333bc6134af10aff7d58876551bf6fdaf0e7" Jul 16 21:54:05.463889 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:05.463865 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs"] Jul 16 21:54:05.474321 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:05.474297 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-xslrs"] Jul 16 21:54:06.450483 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:06.450452 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf21542b-399b-4734-9d44-3622209f237a" path="/var/lib/kubelet/pods/cf21542b-399b-4734-9d44-3622209f237a/volumes" Jul 16 21:54:19.096360 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096330 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5"] Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096652 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c3259b-f997-4ae2-9361-5e48d57700c0" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096662 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c3259b-f997-4ae2-9361-5e48d57700c0" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096676 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c3259b-f997-4ae2-9361-5e48d57700c0" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096681 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c3259b-f997-4ae2-9361-5e48d57700c0" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096691 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11aa1311-51e2-42a4-a6a2-dc2571a456f5" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096697 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11aa1311-51e2-42a4-a6a2-dc2571a456f5" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096702 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf21542b-399b-4734-9d44-3622209f237a" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096707 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf21542b-399b-4734-9d44-3622209f237a" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096714 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c3259b-f997-4ae2-9361-5e48d57700c0" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096719 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c3259b-f997-4ae2-9361-5e48d57700c0" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096726 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf21542b-399b-4734-9d44-3622209f237a" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096731 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf21542b-399b-4734-9d44-3622209f237a" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096738 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11aa1311-51e2-42a4-a6a2-dc2571a456f5" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096742 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11aa1311-51e2-42a4-a6a2-dc2571a456f5" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096749 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096753 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096759 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf21542b-399b-4734-9d44-3622209f237a" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096765 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf21542b-399b-4734-9d44-3622209f237a" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096770 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" containerName="revoke-keys" Jul 16 21:54:19.096798 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096775 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" containerName="revoke-keys" Jul 16 21:54:19.097617 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096824 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="11aa1311-51e2-42a4-a6a2-dc2571a456f5" containerName="revoke-keys" Jul 16 21:54:19.097617 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096832 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf21542b-399b-4734-9d44-3622209f237a" containerName="revoke-keys" Jul 16 21:54:19.097617 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096838 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1c3259b-f997-4ae2-9361-5e48d57700c0" containerName="revoke-keys" Jul 16 21:54:19.097617 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096843 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" containerName="revoke-keys" Jul 16 21:54:19.097617 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096849 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" containerName="revoke-keys" Jul 16 21:54:19.097617 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096854 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="11aa1311-51e2-42a4-a6a2-dc2571a456f5" containerName="revoke-keys" Jul 16 21:54:19.097617 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096860 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1c3259b-f997-4ae2-9361-5e48d57700c0" containerName="revoke-keys" Jul 16 21:54:19.097617 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.096866 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf21542b-399b-4734-9d44-3622209f237a" containerName="revoke-keys" Jul 16 21:54:19.106907 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.106884 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.110646 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.110624 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5b4b2622-openshift-default-dockercfg-sp94s\"" Jul 16 21:54:19.120308 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.120284 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5"] Jul 16 21:54:19.182003 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.181969 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-token\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.182160 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.182040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-credential-socket\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.182160 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.182087 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsl2p\" (UniqueName: \"kubernetes.io/projected/d6ced280-d6ed-407d-9870-ed4d86ccde1d-kube-api-access-tsl2p\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.182160 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.182109 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-envoy\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.182335 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.182202 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-podinfo\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.182335 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.182258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-workload-certs\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.182335 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.182291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istiod-ca-cert\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.182335 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.182318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-workload-socket\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.182335 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.182332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-data\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.283023 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.282980 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-podinfo\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.283240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.283057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-workload-certs\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.283240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.283111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istiod-ca-cert\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.283240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.283148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-workload-socket\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.283240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.283171 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-data\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.283240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.283230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-token\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.283530 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.283363 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-credential-socket\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.283530 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.283438 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsl2p\" (UniqueName: \"kubernetes.io/projected/d6ced280-d6ed-407d-9870-ed4d86ccde1d-kube-api-access-tsl2p\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.283530 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.283473 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-envoy\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.283697 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.283526 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-workload-socket\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.283697 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.283542 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-workload-certs\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.283697 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.283619 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-data\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.283854 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.283832 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-credential-socket\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.283961 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.283922 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istiod-ca-cert\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.285947 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.285924 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-podinfo\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.286074 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.286005 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-envoy\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.295552 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.295527 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsl2p\" (UniqueName: \"kubernetes.io/projected/d6ced280-d6ed-407d-9870-ed4d86ccde1d-kube-api-access-tsl2p\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.297761 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.297738 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-token\") pod \"e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.416821 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.416793 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:19.554718 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.554687 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5"] Jul 16 21:54:19.556126 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:54:19.556097 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ced280_d6ed_407d_9870_ed4d86ccde1d.slice/crio-72c75cbd25b7cd1a88f2a391fbe3a36861a78798562c2f170d2e94ea582fe2c6 WatchSource:0}: Error finding container 72c75cbd25b7cd1a88f2a391fbe3a36861a78798562c2f170d2e94ea582fe2c6: Status 404 returned error can't find the container with id 72c75cbd25b7cd1a88f2a391fbe3a36861a78798562c2f170d2e94ea582fe2c6 Jul 16 21:54:19.558244 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.558211 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:54:19.558313 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.558278 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:54:19.558313 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:19.558308 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:54:20.516785 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:20.516749 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" event={"ID":"d6ced280-d6ed-407d-9870-ed4d86ccde1d","Type":"ContainerStarted","Data":"00a7198557b8e7c483712208e9a0240ce99c26198f3eba0c9f21074bd1d1cf1f"} Jul 16 21:54:20.517144 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:20.516813 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" event={"ID":"d6ced280-d6ed-407d-9870-ed4d86ccde1d","Type":"ContainerStarted","Data":"72c75cbd25b7cd1a88f2a391fbe3a36861a78798562c2f170d2e94ea582fe2c6"} Jul 16 21:54:20.553438 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:20.553387 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" podStartSLOduration=1.553371384 podStartE2EDuration="1.553371384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:20.546426096 +0000 UTC m=+2676.721534314" watchObservedRunningTime="2026-07-16 21:54:20.553371384 +0000 UTC m=+2676.728479576" Jul 16 21:54:21.417352 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:21.417316 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:21.422126 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:21.422102 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:21.521779 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:21.521748 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:21.522747 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:21.522730 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:22.741459 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.741427 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv"] Jul 16 21:54:22.741806 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.741776 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" containerName="revoke-keys" Jul 16 21:54:22.741806 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.741788 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" containerName="revoke-keys" Jul 16 21:54:22.741884 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.741808 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11aa1311-51e2-42a4-a6a2-dc2571a456f5" containerName="revoke-keys" Jul 16 21:54:22.741884 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.741814 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11aa1311-51e2-42a4-a6a2-dc2571a456f5" containerName="revoke-keys" Jul 16 21:54:22.741884 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.741869 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf21542b-399b-4734-9d44-3622209f237a" containerName="revoke-keys" Jul 16 21:54:22.741884 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.741878 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce72c2e6-108c-4aa8-bfb8-5a68ccfbc9d4" containerName="revoke-keys" Jul 16 21:54:22.742007 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.741885 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="11aa1311-51e2-42a4-a6a2-dc2571a456f5" containerName="revoke-keys" Jul 16 21:54:22.742007 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.741892 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1c3259b-f997-4ae2-9361-5e48d57700c0" containerName="revoke-keys" Jul 16 21:54:22.744796 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.744779 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" Jul 16 21:54:22.752455 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.752437 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5b4b2622-serving-cert\"" Jul 16 21:54:22.752831 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.752819 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xcg5c\"" Jul 16 21:54:22.754937 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.754922 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:54:22.756005 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.755990 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 21:54:22.783052 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.783030 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv"] Jul 16 21:54:22.817690 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.817667 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b53099c-a390-4a84-bd57-1560ebfec0c5-openshift-service-ca\") pod \"maas-api-e2e-mt-5b4b2622-7749885687-dkxfv\" (UID: \"1b53099c-a390-4a84-bd57-1560ebfec0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" Jul 16 21:54:22.817800 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.817730 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b53099c-a390-4a84-bd57-1560ebfec0c5-maas-api-tls\") pod \"maas-api-e2e-mt-5b4b2622-7749885687-dkxfv\" (UID: \"1b53099c-a390-4a84-bd57-1560ebfec0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" Jul 16 21:54:22.817800 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.817772 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdl6f\" (UniqueName: \"kubernetes.io/projected/1b53099c-a390-4a84-bd57-1560ebfec0c5-kube-api-access-xdl6f\") pod \"maas-api-e2e-mt-5b4b2622-7749885687-dkxfv\" (UID: \"1b53099c-a390-4a84-bd57-1560ebfec0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" Jul 16 21:54:22.918564 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.918522 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdl6f\" (UniqueName: \"kubernetes.io/projected/1b53099c-a390-4a84-bd57-1560ebfec0c5-kube-api-access-xdl6f\") pod \"maas-api-e2e-mt-5b4b2622-7749885687-dkxfv\" (UID: \"1b53099c-a390-4a84-bd57-1560ebfec0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" Jul 16 21:54:22.918718 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.918590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b53099c-a390-4a84-bd57-1560ebfec0c5-openshift-service-ca\") pod \"maas-api-e2e-mt-5b4b2622-7749885687-dkxfv\" (UID: \"1b53099c-a390-4a84-bd57-1560ebfec0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" Jul 16 21:54:22.918718 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.918651 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b53099c-a390-4a84-bd57-1560ebfec0c5-maas-api-tls\") pod \"maas-api-e2e-mt-5b4b2622-7749885687-dkxfv\" (UID: \"1b53099c-a390-4a84-bd57-1560ebfec0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" Jul 16 21:54:22.919258 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.919236 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b53099c-a390-4a84-bd57-1560ebfec0c5-openshift-service-ca\") pod \"maas-api-e2e-mt-5b4b2622-7749885687-dkxfv\" (UID: \"1b53099c-a390-4a84-bd57-1560ebfec0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" Jul 16 21:54:22.920963 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.920940 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b53099c-a390-4a84-bd57-1560ebfec0c5-maas-api-tls\") pod \"maas-api-e2e-mt-5b4b2622-7749885687-dkxfv\" (UID: \"1b53099c-a390-4a84-bd57-1560ebfec0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" Jul 16 21:54:22.937468 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:22.937445 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdl6f\" (UniqueName: \"kubernetes.io/projected/1b53099c-a390-4a84-bd57-1560ebfec0c5-kube-api-access-xdl6f\") pod \"maas-api-e2e-mt-5b4b2622-7749885687-dkxfv\" (UID: \"1b53099c-a390-4a84-bd57-1560ebfec0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" Jul 16 21:54:23.055273 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:23.055201 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" Jul 16 21:54:23.197799 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:23.197769 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv"] Jul 16 21:54:23.201063 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:54:23.201038 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b53099c_a390_4a84_bd57_1560ebfec0c5.slice/crio-fbb09c65b37673b7fc62b73de13d8bd32745f364547652420b4003a4a533b851 WatchSource:0}: Error finding container fbb09c65b37673b7fc62b73de13d8bd32745f364547652420b4003a4a533b851: Status 404 returned error can't find the container with id fbb09c65b37673b7fc62b73de13d8bd32745f364547652420b4003a4a533b851 Jul 16 21:54:23.532584 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:23.532553 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" event={"ID":"1b53099c-a390-4a84-bd57-1560ebfec0c5","Type":"ContainerStarted","Data":"fbb09c65b37673b7fc62b73de13d8bd32745f364547652420b4003a4a533b851"} Jul 16 21:54:24.539480 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:24.539448 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" event={"ID":"1b53099c-a390-4a84-bd57-1560ebfec0c5","Type":"ContainerStarted","Data":"ee8499cda7214cb1d145e0438694464fafc8986eb385ddceaa668824098b862e"} Jul 16 21:54:24.539866 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:24.539531 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" Jul 16 21:54:24.565502 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:24.565454 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" podStartSLOduration=1.919401975 podStartE2EDuration="2.565440666s" totalImagesPullingTime="646.038691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:24.563259709 +0000 UTC m=+2680.738367911" watchObservedRunningTime="2026-07-16 21:54:24.565440666 +0000 UTC m=+2680.740548857" Jul 16 21:54:30.550400 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:30.550367 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" Jul 16 21:54:33.721225 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.721176 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd"] Jul 16 21:54:33.724758 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.724742 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.727740 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.727708 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5b4b2622\"/\"openshift-service-ca.crt\"" Jul 16 21:54:33.727883 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.727847 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5b4b2622\"/\"kube-root-ca.crt\"" Jul 16 21:54:33.729036 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.729019 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5b4b2622\"/\"default-dockercfg-xhnpc\"" Jul 16 21:54:33.729154 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.729049 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5b4b2622\"/\"e2e-lifecycle-model-5b4b2622-kserve-self-signed-certs\"" Jul 16 21:54:33.736540 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.736516 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd"] Jul 16 21:54:33.814829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.814800 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-dshm\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.814977 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.814847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-tmp-dir\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.814977 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.814927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-home\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.815064 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.814983 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-model-cache\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.815064 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.815010 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce8d3d72-9c24-4b13-b70a-faf085316c16-tls-certs\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.815064 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.815025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khsr2\" (UniqueName: \"kubernetes.io/projected/ce8d3d72-9c24-4b13-b70a-faf085316c16-kube-api-access-khsr2\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.916288 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.916252 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-home\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.916454 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.916309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-model-cache\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.916454 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.916338 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce8d3d72-9c24-4b13-b70a-faf085316c16-tls-certs\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.916454 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.916363 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khsr2\" (UniqueName: \"kubernetes.io/projected/ce8d3d72-9c24-4b13-b70a-faf085316c16-kube-api-access-khsr2\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.916454 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.916424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-dshm\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.916721 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.916476 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-tmp-dir\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.916787 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.916720 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-home\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.916787 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.916723 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-model-cache\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.916787 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.916773 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-tmp-dir\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.918636 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.918610 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-dshm\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.918808 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.918792 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce8d3d72-9c24-4b13-b70a-faf085316c16-tls-certs\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:33.928029 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:33.928007 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khsr2\" (UniqueName: \"kubernetes.io/projected/ce8d3d72-9c24-4b13-b70a-faf085316c16-kube-api-access-khsr2\") pod \"e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:34.034228 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:34.034126 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:34.203322 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:34.203250 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd"] Jul 16 21:54:34.207685 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:54:34.207657 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce8d3d72_9c24_4b13_b70a_faf085316c16.slice/crio-003824d6b6ab86881b7f4601c44499dcb470fb958c13dc9a9bbaaaa7e445177b WatchSource:0}: Error finding container 003824d6b6ab86881b7f4601c44499dcb470fb958c13dc9a9bbaaaa7e445177b: Status 404 returned error can't find the container with id 003824d6b6ab86881b7f4601c44499dcb470fb958c13dc9a9bbaaaa7e445177b Jul 16 21:54:34.598347 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:34.598309 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" event={"ID":"ce8d3d72-9c24-4b13-b70a-faf085316c16","Type":"ContainerStarted","Data":"1a86182bf8ce604c7d614ff68a620dbedae9cbbd9ea3701404457d7ac71b3f5e"} Jul 16 21:54:34.598347 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:34.598356 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" event={"ID":"ce8d3d72-9c24-4b13-b70a-faf085316c16","Type":"ContainerStarted","Data":"003824d6b6ab86881b7f4601c44499dcb470fb958c13dc9a9bbaaaa7e445177b"} Jul 16 21:54:34.618578 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:34.618512 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" podStartSLOduration=1.6184905889999999 podStartE2EDuration="1.618490589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:34.617498852 +0000 UTC m=+2690.792607045" watchObservedRunningTime="2026-07-16 21:54:34.618490589 +0000 UTC m=+2690.793598783" Jul 16 21:54:37.412621 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:37.411896 2577 patch_prober.go:28] interesting pod/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": read tcp 10.134.0.2:39832->10.134.0.100:15021: read: connection reset by peer" start-of-body= Jul 16 21:54:37.412621 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:37.411978 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" podUID="d6ced280-d6ed-407d-9870-ed4d86ccde1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": read tcp 10.134.0.2:39832->10.134.0.100:15021: read: connection reset by peer" Jul 16 21:54:37.617982 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:37.617902 2577 generic.go:356] "Generic (PLEG): container finished" podID="d6ced280-d6ed-407d-9870-ed4d86ccde1d" containerID="00a7198557b8e7c483712208e9a0240ce99c26198f3eba0c9f21074bd1d1cf1f" exitCode=0 Jul 16 21:54:37.617982 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:37.617938 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" event={"ID":"d6ced280-d6ed-407d-9870-ed4d86ccde1d","Type":"ContainerDied","Data":"00a7198557b8e7c483712208e9a0240ce99c26198f3eba0c9f21074bd1d1cf1f"} Jul 16 21:54:37.618419 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:37.618400 2577 scope.go:117] "RemoveContainer" containerID="00a7198557b8e7c483712208e9a0240ce99c26198f3eba0c9f21074bd1d1cf1f" Jul 16 21:54:37.619434 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:37.619404 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:54:37.619538 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:37.619470 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:54:37.619538 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:37.619499 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:54:38.625240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:38.625203 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" event={"ID":"d6ced280-d6ed-407d-9870-ed4d86ccde1d","Type":"ContainerStarted","Data":"7552f922111469029ada975d73a8d5def4b2fe6459c545d778c7cdc074722059"} Jul 16 21:54:39.417771 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:39.417733 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:39.417974 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:39.417917 2577 patch_prober.go:28] interesting pod/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Jul 16 21:54:39.418051 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:39.417968 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" podUID="d6ced280-d6ed-407d-9870-ed4d86ccde1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Jul 16 21:54:40.417807 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:40.417775 2577 patch_prober.go:28] interesting pod/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Jul 16 21:54:40.418172 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:40.417828 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" podUID="d6ced280-d6ed-407d-9870-ed4d86ccde1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Jul 16 21:54:41.417517 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:41.417479 2577 patch_prober.go:28] interesting pod/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Jul 16 21:54:41.417699 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:41.417531 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" podUID="d6ced280-d6ed-407d-9870-ed4d86ccde1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Jul 16 21:54:42.450744 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:42.450717 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:42.451148 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:42.450940 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:42.451658 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:42.451643 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:54:44.034681 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:44.034651 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:44.035073 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:44.034690 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:44.046916 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:44.046893 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:54:44.505991 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:44.505965 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:54:44.513836 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:44.513814 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:54:44.668043 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:54:44.668019 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:55:16.056797 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.056764 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv"] Jul 16 21:55:16.057378 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.056953 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" podUID="1b53099c-a390-4a84-bd57-1560ebfec0c5" containerName="maas-api" containerID="cri-o://ee8499cda7214cb1d145e0438694464fafc8986eb385ddceaa668824098b862e" gracePeriod=30 Jul 16 21:55:16.307226 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.307150 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" Jul 16 21:55:16.377359 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.377333 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdl6f\" (UniqueName: \"kubernetes.io/projected/1b53099c-a390-4a84-bd57-1560ebfec0c5-kube-api-access-xdl6f\") pod \"1b53099c-a390-4a84-bd57-1560ebfec0c5\" (UID: \"1b53099c-a390-4a84-bd57-1560ebfec0c5\") " Jul 16 21:55:16.377529 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.377373 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b53099c-a390-4a84-bd57-1560ebfec0c5-openshift-service-ca\") pod \"1b53099c-a390-4a84-bd57-1560ebfec0c5\" (UID: \"1b53099c-a390-4a84-bd57-1560ebfec0c5\") " Jul 16 21:55:16.377529 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.377398 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b53099c-a390-4a84-bd57-1560ebfec0c5-maas-api-tls\") pod \"1b53099c-a390-4a84-bd57-1560ebfec0c5\" (UID: \"1b53099c-a390-4a84-bd57-1560ebfec0c5\") " Jul 16 21:55:16.377702 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.377677 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b53099c-a390-4a84-bd57-1560ebfec0c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1b53099c-a390-4a84-bd57-1560ebfec0c5" (UID: "1b53099c-a390-4a84-bd57-1560ebfec0c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:55:16.379423 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.379393 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b53099c-a390-4a84-bd57-1560ebfec0c5-kube-api-access-xdl6f" (OuterVolumeSpecName: "kube-api-access-xdl6f") pod "1b53099c-a390-4a84-bd57-1560ebfec0c5" (UID: "1b53099c-a390-4a84-bd57-1560ebfec0c5"). InnerVolumeSpecName "kube-api-access-xdl6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:55:16.379423 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.379410 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b53099c-a390-4a84-bd57-1560ebfec0c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1b53099c-a390-4a84-bd57-1560ebfec0c5" (UID: "1b53099c-a390-4a84-bd57-1560ebfec0c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:55:16.478816 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.478783 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdl6f\" (UniqueName: \"kubernetes.io/projected/1b53099c-a390-4a84-bd57-1560ebfec0c5-kube-api-access-xdl6f\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:55:16.478816 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.478815 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b53099c-a390-4a84-bd57-1560ebfec0c5-openshift-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:55:16.478816 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.478824 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b53099c-a390-4a84-bd57-1560ebfec0c5-maas-api-tls\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:55:16.822452 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.822419 2577 generic.go:356] "Generic (PLEG): container finished" podID="1b53099c-a390-4a84-bd57-1560ebfec0c5" containerID="ee8499cda7214cb1d145e0438694464fafc8986eb385ddceaa668824098b862e" exitCode=0 Jul 16 21:55:16.822622 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.822462 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" Jul 16 21:55:16.822622 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.822460 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" event={"ID":"1b53099c-a390-4a84-bd57-1560ebfec0c5","Type":"ContainerDied","Data":"ee8499cda7214cb1d145e0438694464fafc8986eb385ddceaa668824098b862e"} Jul 16 21:55:16.822622 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.822600 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv" event={"ID":"1b53099c-a390-4a84-bd57-1560ebfec0c5","Type":"ContainerDied","Data":"fbb09c65b37673b7fc62b73de13d8bd32745f364547652420b4003a4a533b851"} Jul 16 21:55:16.822809 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.822640 2577 scope.go:117] "RemoveContainer" containerID="ee8499cda7214cb1d145e0438694464fafc8986eb385ddceaa668824098b862e" Jul 16 21:55:16.831234 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.831218 2577 scope.go:117] "RemoveContainer" containerID="ee8499cda7214cb1d145e0438694464fafc8986eb385ddceaa668824098b862e" Jul 16 21:55:16.831472 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:55:16.831456 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee8499cda7214cb1d145e0438694464fafc8986eb385ddceaa668824098b862e\": container with ID starting with ee8499cda7214cb1d145e0438694464fafc8986eb385ddceaa668824098b862e not found: ID does not exist" containerID="ee8499cda7214cb1d145e0438694464fafc8986eb385ddceaa668824098b862e" Jul 16 21:55:16.831516 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.831481 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee8499cda7214cb1d145e0438694464fafc8986eb385ddceaa668824098b862e"} err="failed to get container status \"ee8499cda7214cb1d145e0438694464fafc8986eb385ddceaa668824098b862e\": rpc error: code = NotFound desc = could not find container \"ee8499cda7214cb1d145e0438694464fafc8986eb385ddceaa668824098b862e\": container with ID starting with ee8499cda7214cb1d145e0438694464fafc8986eb385ddceaa668824098b862e not found: ID does not exist" Jul 16 21:55:16.848045 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.848024 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv"] Jul 16 21:55:16.856294 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:16.856270 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b4b2622-7749885687-dkxfv"] Jul 16 21:55:18.451201 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:18.451158 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b53099c-a390-4a84-bd57-1560ebfec0c5" path="/var/lib/kubelet/pods/1b53099c-a390-4a84-bd57-1560ebfec0c5/volumes" Jul 16 21:55:25.271249 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:25.271215 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd"] Jul 16 21:55:40.291395 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.291328 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" podUID="ce8d3d72-9c24-4b13-b70a-faf085316c16" containerName="main" containerID="cri-o://1a86182bf8ce604c7d614ff68a620dbedae9cbbd9ea3701404457d7ac71b3f5e" gracePeriod=45 Jul 16 21:55:40.532944 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.532920 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:55:40.672391 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.672359 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-tmp-dir\") pod \"ce8d3d72-9c24-4b13-b70a-faf085316c16\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " Jul 16 21:55:40.672579 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.672409 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khsr2\" (UniqueName: \"kubernetes.io/projected/ce8d3d72-9c24-4b13-b70a-faf085316c16-kube-api-access-khsr2\") pod \"ce8d3d72-9c24-4b13-b70a-faf085316c16\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " Jul 16 21:55:40.672579 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.672452 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-home\") pod \"ce8d3d72-9c24-4b13-b70a-faf085316c16\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " Jul 16 21:55:40.672579 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.672473 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-dshm\") pod \"ce8d3d72-9c24-4b13-b70a-faf085316c16\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " Jul 16 21:55:40.672579 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.672496 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-model-cache\") pod \"ce8d3d72-9c24-4b13-b70a-faf085316c16\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " Jul 16 21:55:40.672579 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.672512 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce8d3d72-9c24-4b13-b70a-faf085316c16-tls-certs\") pod \"ce8d3d72-9c24-4b13-b70a-faf085316c16\" (UID: \"ce8d3d72-9c24-4b13-b70a-faf085316c16\") " Jul 16 21:55:40.672824 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.672690 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce8d3d72-9c24-4b13-b70a-faf085316c16" (UID: "ce8d3d72-9c24-4b13-b70a-faf085316c16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:55:40.672824 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.672736 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-home" (OuterVolumeSpecName: "home") pod "ce8d3d72-9c24-4b13-b70a-faf085316c16" (UID: "ce8d3d72-9c24-4b13-b70a-faf085316c16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:55:40.672824 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.672792 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce8d3d72-9c24-4b13-b70a-faf085316c16" (UID: "ce8d3d72-9c24-4b13-b70a-faf085316c16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:55:40.674721 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.674696 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8d3d72-9c24-4b13-b70a-faf085316c16-kube-api-access-khsr2" (OuterVolumeSpecName: "kube-api-access-khsr2") pod "ce8d3d72-9c24-4b13-b70a-faf085316c16" (UID: "ce8d3d72-9c24-4b13-b70a-faf085316c16"). InnerVolumeSpecName "kube-api-access-khsr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:55:40.674721 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.674712 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-dshm" (OuterVolumeSpecName: "dshm") pod "ce8d3d72-9c24-4b13-b70a-faf085316c16" (UID: "ce8d3d72-9c24-4b13-b70a-faf085316c16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:55:40.674873 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.674695 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce8d3d72-9c24-4b13-b70a-faf085316c16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce8d3d72-9c24-4b13-b70a-faf085316c16" (UID: "ce8d3d72-9c24-4b13-b70a-faf085316c16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:55:40.773544 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.773518 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-home\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:55:40.773544 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.773539 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-dshm\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:55:40.773544 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.773549 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-model-cache\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:55:40.773748 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.773558 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce8d3d72-9c24-4b13-b70a-faf085316c16-tls-certs\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:55:40.773748 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.773566 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3d72-9c24-4b13-b70a-faf085316c16-tmp-dir\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:55:40.773748 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.773574 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khsr2\" (UniqueName: \"kubernetes.io/projected/ce8d3d72-9c24-4b13-b70a-faf085316c16-kube-api-access-khsr2\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:55:40.950441 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.950355 2577 generic.go:356] "Generic (PLEG): container finished" podID="ce8d3d72-9c24-4b13-b70a-faf085316c16" containerID="1a86182bf8ce604c7d614ff68a620dbedae9cbbd9ea3701404457d7ac71b3f5e" exitCode=0 Jul 16 21:55:40.950441 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.950399 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" event={"ID":"ce8d3d72-9c24-4b13-b70a-faf085316c16","Type":"ContainerDied","Data":"1a86182bf8ce604c7d614ff68a620dbedae9cbbd9ea3701404457d7ac71b3f5e"} Jul 16 21:55:40.950441 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.950419 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" Jul 16 21:55:40.950441 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.950431 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd" event={"ID":"ce8d3d72-9c24-4b13-b70a-faf085316c16","Type":"ContainerDied","Data":"003824d6b6ab86881b7f4601c44499dcb470fb958c13dc9a9bbaaaa7e445177b"} Jul 16 21:55:40.950720 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.950452 2577 scope.go:117] "RemoveContainer" containerID="1a86182bf8ce604c7d614ff68a620dbedae9cbbd9ea3701404457d7ac71b3f5e" Jul 16 21:55:40.962003 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.961988 2577 scope.go:117] "RemoveContainer" containerID="1a86182bf8ce604c7d614ff68a620dbedae9cbbd9ea3701404457d7ac71b3f5e" Jul 16 21:55:40.962252 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:55:40.962231 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a86182bf8ce604c7d614ff68a620dbedae9cbbd9ea3701404457d7ac71b3f5e\": container with ID starting with 1a86182bf8ce604c7d614ff68a620dbedae9cbbd9ea3701404457d7ac71b3f5e not found: ID does not exist" containerID="1a86182bf8ce604c7d614ff68a620dbedae9cbbd9ea3701404457d7ac71b3f5e" Jul 16 21:55:40.962311 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.962261 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a86182bf8ce604c7d614ff68a620dbedae9cbbd9ea3701404457d7ac71b3f5e"} err="failed to get container status \"1a86182bf8ce604c7d614ff68a620dbedae9cbbd9ea3701404457d7ac71b3f5e\": rpc error: code = NotFound desc = could not find container \"1a86182bf8ce604c7d614ff68a620dbedae9cbbd9ea3701404457d7ac71b3f5e\": container with ID starting with 1a86182bf8ce604c7d614ff68a620dbedae9cbbd9ea3701404457d7ac71b3f5e not found: ID does not exist" Jul 16 21:55:40.973156 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.973133 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd"] Jul 16 21:55:40.976498 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:40.976478 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5b4b2622/e2e-lifecycle-model-5b4b2622-kserve-6f57b58d87-ll2kd"] Jul 16 21:55:42.450779 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:42.450750 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce8d3d72-9c24-4b13-b70a-faf085316c16" path="/var/lib/kubelet/pods/ce8d3d72-9c24-4b13-b70a-faf085316c16/volumes" Jul 16 21:55:45.336812 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.336780 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn"] Jul 16 21:55:45.337223 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.337164 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b53099c-a390-4a84-bd57-1560ebfec0c5" containerName="maas-api" Jul 16 21:55:45.337223 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.337178 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b53099c-a390-4a84-bd57-1560ebfec0c5" containerName="maas-api" Jul 16 21:55:45.337223 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.337202 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce8d3d72-9c24-4b13-b70a-faf085316c16" containerName="main" Jul 16 21:55:45.337223 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.337209 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce8d3d72-9c24-4b13-b70a-faf085316c16" containerName="main" Jul 16 21:55:45.337372 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.337267 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b53099c-a390-4a84-bd57-1560ebfec0c5" containerName="maas-api" Jul 16 21:55:45.337372 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.337275 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce8d3d72-9c24-4b13-b70a-faf085316c16" containerName="main" Jul 16 21:55:45.341406 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.341392 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" Jul 16 21:55:45.344240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.344214 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:55:45.344341 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.344259 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:55:45.350276 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.350252 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn"] Jul 16 21:55:45.515923 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.515883 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e9da13-7de0-4bcf-8d7f-9bbab041300e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn\" (UID: \"c7e9da13-7de0-4bcf-8d7f-9bbab041300e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" Jul 16 21:55:45.616842 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.616755 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e9da13-7de0-4bcf-8d7f-9bbab041300e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn\" (UID: \"c7e9da13-7de0-4bcf-8d7f-9bbab041300e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" Jul 16 21:55:45.617470 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.617443 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e9da13-7de0-4bcf-8d7f-9bbab041300e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn\" (UID: \"c7e9da13-7de0-4bcf-8d7f-9bbab041300e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" Jul 16 21:55:45.656892 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.656857 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" Jul 16 21:55:45.797411 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.797386 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn"] Jul 16 21:55:45.799464 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:55:45.799437 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e9da13_7de0_4bcf_8d7f_9bbab041300e.slice/crio-33211cbe463b0ad167de9942c61dc813253ad816b5d7fd9102d6c6a170d21699 WatchSource:0}: Error finding container 33211cbe463b0ad167de9942c61dc813253ad816b5d7fd9102d6c6a170d21699: Status 404 returned error can't find the container with id 33211cbe463b0ad167de9942c61dc813253ad816b5d7fd9102d6c6a170d21699 Jul 16 21:55:45.976813 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:45.976786 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" event={"ID":"c7e9da13-7de0-4bcf-8d7f-9bbab041300e","Type":"ContainerStarted","Data":"33211cbe463b0ad167de9942c61dc813253ad816b5d7fd9102d6c6a170d21699"} Jul 16 21:55:46.983117 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:46.983085 2577 generic.go:356] "Generic (PLEG): container finished" podID="c7e9da13-7de0-4bcf-8d7f-9bbab041300e" containerID="1ecf4df80be4890001994d1198a04b781a7a47603a1cf32bee1687fe36f8e842" exitCode=6 Jul 16 21:55:46.983502 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:46.983178 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" event={"ID":"c7e9da13-7de0-4bcf-8d7f-9bbab041300e","Type":"ContainerDied","Data":"1ecf4df80be4890001994d1198a04b781a7a47603a1cf32bee1687fe36f8e842"} Jul 16 21:55:46.983502 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:46.983331 2577 scope.go:117] "RemoveContainer" containerID="1ecf4df80be4890001994d1198a04b781a7a47603a1cf32bee1687fe36f8e842" Jul 16 21:55:47.989975 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:47.989940 2577 generic.go:356] "Generic (PLEG): container finished" podID="c7e9da13-7de0-4bcf-8d7f-9bbab041300e" containerID="343227088b09f2d9f1e61de11b4d5f88d1315e0e7fbe871bf28d886301332454" exitCode=6 Jul 16 21:55:47.990458 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:47.989980 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" event={"ID":"c7e9da13-7de0-4bcf-8d7f-9bbab041300e","Type":"ContainerDied","Data":"343227088b09f2d9f1e61de11b4d5f88d1315e0e7fbe871bf28d886301332454"} Jul 16 21:55:47.990458 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:47.990031 2577 scope.go:117] "RemoveContainer" containerID="1ecf4df80be4890001994d1198a04b781a7a47603a1cf32bee1687fe36f8e842" Jul 16 21:55:47.990458 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:47.990211 2577 scope.go:117] "RemoveContainer" containerID="343227088b09f2d9f1e61de11b4d5f88d1315e0e7fbe871bf28d886301332454" Jul 16 21:55:47.990591 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:55:47.990452 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn_odh-ai-gateway-infra(c7e9da13-7de0-4bcf-8d7f-9bbab041300e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" podUID="c7e9da13-7de0-4bcf-8d7f-9bbab041300e" Jul 16 21:55:48.997479 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:55:48.997454 2577 scope.go:117] "RemoveContainer" containerID="343227088b09f2d9f1e61de11b4d5f88d1315e0e7fbe871bf28d886301332454" Jul 16 21:55:48.997855 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:55:48.997630 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn_odh-ai-gateway-infra(c7e9da13-7de0-4bcf-8d7f-9bbab041300e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" podUID="c7e9da13-7de0-4bcf-8d7f-9bbab041300e" Jul 16 21:56:00.445373 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:00.445342 2577 scope.go:117] "RemoveContainer" containerID="343227088b09f2d9f1e61de11b4d5f88d1315e0e7fbe871bf28d886301332454" Jul 16 21:56:01.060316 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:01.060283 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" event={"ID":"c7e9da13-7de0-4bcf-8d7f-9bbab041300e","Type":"ContainerStarted","Data":"ac137e137efd0fba3be4f8f09855045d752c2bdea0edfd7f735f7b8300f1d0ec"} Jul 16 21:56:01.080367 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:01.080319 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" podStartSLOduration=16.08030365 podStartE2EDuration="16.08030365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:01.078418349 +0000 UTC m=+2777.253526542" watchObservedRunningTime="2026-07-16 21:56:01.08030365 +0000 UTC m=+2777.255411842" Jul 16 21:56:02.066765 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:02.066733 2577 generic.go:356] "Generic (PLEG): container finished" podID="c7e9da13-7de0-4bcf-8d7f-9bbab041300e" containerID="ac137e137efd0fba3be4f8f09855045d752c2bdea0edfd7f735f7b8300f1d0ec" exitCode=6 Jul 16 21:56:02.066765 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:02.066769 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" event={"ID":"c7e9da13-7de0-4bcf-8d7f-9bbab041300e","Type":"ContainerDied","Data":"ac137e137efd0fba3be4f8f09855045d752c2bdea0edfd7f735f7b8300f1d0ec"} Jul 16 21:56:02.067257 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:02.066825 2577 scope.go:117] "RemoveContainer" containerID="343227088b09f2d9f1e61de11b4d5f88d1315e0e7fbe871bf28d886301332454" Jul 16 21:56:02.067257 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:02.067110 2577 scope.go:117] "RemoveContainer" containerID="ac137e137efd0fba3be4f8f09855045d752c2bdea0edfd7f735f7b8300f1d0ec" Jul 16 21:56:02.067343 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:56:02.067325 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn_odh-ai-gateway-infra(c7e9da13-7de0-4bcf-8d7f-9bbab041300e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" podUID="c7e9da13-7de0-4bcf-8d7f-9bbab041300e" Jul 16 21:56:02.095706 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:02.095684 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn"] Jul 16 21:56:02.607767 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:02.607736 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5"] Jul 16 21:56:02.608038 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:02.608004 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" podUID="d6ced280-d6ed-407d-9870-ed4d86ccde1d" containerName="istio-proxy" containerID="cri-o://7552f922111469029ada975d73a8d5def4b2fe6459c545d778c7cdc074722059" gracePeriod=30 Jul 16 21:56:03.205068 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:03.205045 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" Jul 16 21:56:03.259928 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:03.259891 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e9da13-7de0-4bcf-8d7f-9bbab041300e-maas-api-service-ca\") pod \"c7e9da13-7de0-4bcf-8d7f-9bbab041300e\" (UID: \"c7e9da13-7de0-4bcf-8d7f-9bbab041300e\") " Jul 16 21:56:03.260260 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:03.260239 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e9da13-7de0-4bcf-8d7f-9bbab041300e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c7e9da13-7de0-4bcf-8d7f-9bbab041300e" (UID: "c7e9da13-7de0-4bcf-8d7f-9bbab041300e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:56:03.361482 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:03.361412 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e9da13-7de0-4bcf-8d7f-9bbab041300e-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:56:04.080022 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:04.079984 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" event={"ID":"c7e9da13-7de0-4bcf-8d7f-9bbab041300e","Type":"ContainerDied","Data":"33211cbe463b0ad167de9942c61dc813253ad816b5d7fd9102d6c6a170d21699"} Jul 16 21:56:04.080022 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:04.080024 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn" Jul 16 21:56:04.080313 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:04.080043 2577 scope.go:117] "RemoveContainer" containerID="ac137e137efd0fba3be4f8f09855045d752c2bdea0edfd7f735f7b8300f1d0ec" Jul 16 21:56:04.103640 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:04.103617 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn"] Jul 16 21:56:04.109969 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:04.109938 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-j6kfn"] Jul 16 21:56:04.450236 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:04.450207 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7e9da13-7de0-4bcf-8d7f-9bbab041300e" path="/var/lib/kubelet/pods/c7e9da13-7de0-4bcf-8d7f-9bbab041300e/volumes" Jul 16 21:56:07.857751 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.857725 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:56:07.898239 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.898217 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-workload-certs\") pod \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " Jul 16 21:56:07.898381 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.898244 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-data\") pod \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " Jul 16 21:56:07.898381 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.898277 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-token\") pod \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " Jul 16 21:56:07.898381 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.898359 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-credential-socket\") pod \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " Jul 16 21:56:07.898554 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.898396 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istiod-ca-cert\") pod \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " Jul 16 21:56:07.898554 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.898443 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-envoy\") pod \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " Jul 16 21:56:07.898554 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.898472 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsl2p\" (UniqueName: \"kubernetes.io/projected/d6ced280-d6ed-407d-9870-ed4d86ccde1d-kube-api-access-tsl2p\") pod \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " Jul 16 21:56:07.898554 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.898502 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-workload-socket\") pod \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " Jul 16 21:56:07.898554 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.898505 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d6ced280-d6ed-407d-9870-ed4d86ccde1d" (UID: "d6ced280-d6ed-407d-9870-ed4d86ccde1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:56:07.898554 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.898545 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-podinfo\") pod \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\" (UID: \"d6ced280-d6ed-407d-9870-ed4d86ccde1d\") " Jul 16 21:56:07.898884 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.898670 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d6ced280-d6ed-407d-9870-ed4d86ccde1d" (UID: "d6ced280-d6ed-407d-9870-ed4d86ccde1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:56:07.898884 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.898789 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d6ced280-d6ed-407d-9870-ed4d86ccde1d" (UID: "d6ced280-d6ed-407d-9870-ed4d86ccde1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:56:07.899059 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.898908 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d6ced280-d6ed-407d-9870-ed4d86ccde1d" (UID: "d6ced280-d6ed-407d-9870-ed4d86ccde1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:56:07.899059 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.898992 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-workload-certs\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:56:07.899059 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.899017 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-credential-socket\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:56:07.899059 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.899053 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istiod-ca-cert\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:56:07.899345 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.899293 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-data" (OuterVolumeSpecName: "istio-data") pod "d6ced280-d6ed-407d-9870-ed4d86ccde1d" (UID: "d6ced280-d6ed-407d-9870-ed4d86ccde1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:56:07.900949 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.900923 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-token" (OuterVolumeSpecName: "istio-token") pod "d6ced280-d6ed-407d-9870-ed4d86ccde1d" (UID: "d6ced280-d6ed-407d-9870-ed4d86ccde1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:56:07.901044 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.900949 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ced280-d6ed-407d-9870-ed4d86ccde1d-kube-api-access-tsl2p" (OuterVolumeSpecName: "kube-api-access-tsl2p") pod "d6ced280-d6ed-407d-9870-ed4d86ccde1d" (UID: "d6ced280-d6ed-407d-9870-ed4d86ccde1d"). InnerVolumeSpecName "kube-api-access-tsl2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:56:07.901206 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.901168 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d6ced280-d6ed-407d-9870-ed4d86ccde1d" (UID: "d6ced280-d6ed-407d-9870-ed4d86ccde1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 21:56:07.901276 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:07.901203 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d6ced280-d6ed-407d-9870-ed4d86ccde1d" (UID: "d6ced280-d6ed-407d-9870-ed4d86ccde1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:56:08.000384 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.000360 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-envoy\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:56:08.000384 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.000380 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsl2p\" (UniqueName: \"kubernetes.io/projected/d6ced280-d6ed-407d-9870-ed4d86ccde1d-kube-api-access-tsl2p\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:56:08.000384 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.000389 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-workload-socket\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:56:08.000556 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.000397 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-podinfo\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:56:08.000556 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.000405 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-data\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:56:08.000556 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.000413 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ced280-d6ed-407d-9870-ed4d86ccde1d-istio-token\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:56:08.103003 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.102963 2577 generic.go:356] "Generic (PLEG): container finished" podID="d6ced280-d6ed-407d-9870-ed4d86ccde1d" containerID="7552f922111469029ada975d73a8d5def4b2fe6459c545d778c7cdc074722059" exitCode=0 Jul 16 21:56:08.103142 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.103012 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" Jul 16 21:56:08.103142 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.103049 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" event={"ID":"d6ced280-d6ed-407d-9870-ed4d86ccde1d","Type":"ContainerDied","Data":"7552f922111469029ada975d73a8d5def4b2fe6459c545d778c7cdc074722059"} Jul 16 21:56:08.103142 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.103099 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5" event={"ID":"d6ced280-d6ed-407d-9870-ed4d86ccde1d","Type":"ContainerDied","Data":"72c75cbd25b7cd1a88f2a391fbe3a36861a78798562c2f170d2e94ea582fe2c6"} Jul 16 21:56:08.103142 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.103116 2577 scope.go:117] "RemoveContainer" containerID="7552f922111469029ada975d73a8d5def4b2fe6459c545d778c7cdc074722059" Jul 16 21:56:08.112415 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.112394 2577 scope.go:117] "RemoveContainer" containerID="00a7198557b8e7c483712208e9a0240ce99c26198f3eba0c9f21074bd1d1cf1f" Jul 16 21:56:08.120130 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.120114 2577 scope.go:117] "RemoveContainer" containerID="7552f922111469029ada975d73a8d5def4b2fe6459c545d778c7cdc074722059" Jul 16 21:56:08.120375 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:56:08.120357 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7552f922111469029ada975d73a8d5def4b2fe6459c545d778c7cdc074722059\": container with ID starting with 7552f922111469029ada975d73a8d5def4b2fe6459c545d778c7cdc074722059 not found: ID does not exist" containerID="7552f922111469029ada975d73a8d5def4b2fe6459c545d778c7cdc074722059" Jul 16 21:56:08.120455 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.120386 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7552f922111469029ada975d73a8d5def4b2fe6459c545d778c7cdc074722059"} err="failed to get container status \"7552f922111469029ada975d73a8d5def4b2fe6459c545d778c7cdc074722059\": rpc error: code = NotFound desc = could not find container \"7552f922111469029ada975d73a8d5def4b2fe6459c545d778c7cdc074722059\": container with ID starting with 7552f922111469029ada975d73a8d5def4b2fe6459c545d778c7cdc074722059 not found: ID does not exist" Jul 16 21:56:08.120455 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.120407 2577 scope.go:117] "RemoveContainer" containerID="00a7198557b8e7c483712208e9a0240ce99c26198f3eba0c9f21074bd1d1cf1f" Jul 16 21:56:08.120672 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:56:08.120651 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00a7198557b8e7c483712208e9a0240ce99c26198f3eba0c9f21074bd1d1cf1f\": container with ID starting with 00a7198557b8e7c483712208e9a0240ce99c26198f3eba0c9f21074bd1d1cf1f not found: ID does not exist" containerID="00a7198557b8e7c483712208e9a0240ce99c26198f3eba0c9f21074bd1d1cf1f" Jul 16 21:56:08.120712 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.120679 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00a7198557b8e7c483712208e9a0240ce99c26198f3eba0c9f21074bd1d1cf1f"} err="failed to get container status \"00a7198557b8e7c483712208e9a0240ce99c26198f3eba0c9f21074bd1d1cf1f\": rpc error: code = NotFound desc = could not find container \"00a7198557b8e7c483712208e9a0240ce99c26198f3eba0c9f21074bd1d1cf1f\": container with ID starting with 00a7198557b8e7c483712208e9a0240ce99c26198f3eba0c9f21074bd1d1cf1f not found: ID does not exist" Jul 16 21:56:08.138660 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.138641 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5"] Jul 16 21:56:08.141984 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.141963 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5b4b2622-openshift-default-57d8fd9887-n4dz5"] Jul 16 21:56:08.450838 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:08.450810 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6ced280-d6ed-407d-9870-ed4d86ccde1d" path="/var/lib/kubelet/pods/d6ced280-d6ed-407d-9870-ed4d86ccde1d/volumes" Jul 16 21:56:13.651882 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.651843 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9"] Jul 16 21:56:13.652297 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.652219 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ced280-d6ed-407d-9870-ed4d86ccde1d" containerName="istio-proxy" Jul 16 21:56:13.652297 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.652235 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ced280-d6ed-407d-9870-ed4d86ccde1d" containerName="istio-proxy" Jul 16 21:56:13.652297 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.652252 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ced280-d6ed-407d-9870-ed4d86ccde1d" containerName="istio-proxy" Jul 16 21:56:13.652297 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.652258 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ced280-d6ed-407d-9870-ed4d86ccde1d" containerName="istio-proxy" Jul 16 21:56:13.652297 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.652264 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e9da13-7de0-4bcf-8d7f-9bbab041300e" containerName="revoke-keys" Jul 16 21:56:13.652297 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.652270 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e9da13-7de0-4bcf-8d7f-9bbab041300e" containerName="revoke-keys" Jul 16 21:56:13.652297 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.652276 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e9da13-7de0-4bcf-8d7f-9bbab041300e" containerName="revoke-keys" Jul 16 21:56:13.652297 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.652281 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e9da13-7de0-4bcf-8d7f-9bbab041300e" containerName="revoke-keys" Jul 16 21:56:13.652297 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.652288 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e9da13-7de0-4bcf-8d7f-9bbab041300e" containerName="revoke-keys" Jul 16 21:56:13.652297 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.652293 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e9da13-7de0-4bcf-8d7f-9bbab041300e" containerName="revoke-keys" Jul 16 21:56:13.652664 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.652348 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e9da13-7de0-4bcf-8d7f-9bbab041300e" containerName="revoke-keys" Jul 16 21:56:13.652664 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.652355 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ced280-d6ed-407d-9870-ed4d86ccde1d" containerName="istio-proxy" Jul 16 21:56:13.652664 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.652365 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e9da13-7de0-4bcf-8d7f-9bbab041300e" containerName="revoke-keys" Jul 16 21:56:13.656678 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.656661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9" Jul 16 21:56:13.661095 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.661076 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:56:13.661262 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.661244 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:56:13.674663 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.674618 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9"] Jul 16 21:56:13.748441 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.748410 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920acc96-c6c8-4424-a663-4b994883cd58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9\" (UID: \"920acc96-c6c8-4424-a663-4b994883cd58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9" Jul 16 21:56:13.849387 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.849351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920acc96-c6c8-4424-a663-4b994883cd58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9\" (UID: \"920acc96-c6c8-4424-a663-4b994883cd58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9" Jul 16 21:56:13.849945 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.849927 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920acc96-c6c8-4424-a663-4b994883cd58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9\" (UID: \"920acc96-c6c8-4424-a663-4b994883cd58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9" Jul 16 21:56:13.967283 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:13.967212 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9" Jul 16 21:56:14.104834 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:14.104807 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9"] Jul 16 21:56:14.106148 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:56:14.106124 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920acc96_c6c8_4424_a663_4b994883cd58.slice/crio-57fe2e09776c755400553b15d08e81afe128b60e113bb8cfc63bf2eab447cf4e WatchSource:0}: Error finding container 57fe2e09776c755400553b15d08e81afe128b60e113bb8cfc63bf2eab447cf4e: Status 404 returned error can't find the container with id 57fe2e09776c755400553b15d08e81afe128b60e113bb8cfc63bf2eab447cf4e Jul 16 21:56:14.135566 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:14.135539 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9" event={"ID":"920acc96-c6c8-4424-a663-4b994883cd58","Type":"ContainerStarted","Data":"57fe2e09776c755400553b15d08e81afe128b60e113bb8cfc63bf2eab447cf4e"} Jul 16 21:56:15.141683 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:15.141651 2577 generic.go:356] "Generic (PLEG): container finished" podID="920acc96-c6c8-4424-a663-4b994883cd58" containerID="0fb216a341e73990938578a7e586ab70fcd9a2a89420537891248e445db0cc60" exitCode=6 Jul 16 21:56:15.142077 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:15.141735 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9" event={"ID":"920acc96-c6c8-4424-a663-4b994883cd58","Type":"ContainerDied","Data":"0fb216a341e73990938578a7e586ab70fcd9a2a89420537891248e445db0cc60"} Jul 16 21:56:15.142077 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:15.141878 2577 scope.go:117] "RemoveContainer" containerID="0fb216a341e73990938578a7e586ab70fcd9a2a89420537891248e445db0cc60" Jul 16 21:56:16.016100 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:16.016067 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9"] Jul 16 21:56:16.016546 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:16.016533 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e9da13-7de0-4bcf-8d7f-9bbab041300e" containerName="revoke-keys" Jul 16 21:56:16.016588 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:16.016548 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ced280-d6ed-407d-9870-ed4d86ccde1d" containerName="istio-proxy" Jul 16 21:56:16.019504 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:16.019489 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9" Jul 16 21:56:16.044325 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:16.044297 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9"] Jul 16 21:56:16.065296 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:16.065268 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a885791-e3fc-4c2f-96d0-2eaec48b1ff4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9\" (UID: \"3a885791-e3fc-4c2f-96d0-2eaec48b1ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9" Jul 16 21:56:16.149721 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:16.149625 2577 generic.go:356] "Generic (PLEG): container finished" podID="920acc96-c6c8-4424-a663-4b994883cd58" containerID="0ccaa8e29bef6bbb701f3fba8ea7228a2445598fdc1b27d24b8c9553b97c7be6" exitCode=6 Jul 16 21:56:16.149721 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:16.149687 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9" event={"ID":"920acc96-c6c8-4424-a663-4b994883cd58","Type":"ContainerDied","Data":"0ccaa8e29bef6bbb701f3fba8ea7228a2445598fdc1b27d24b8c9553b97c7be6"} Jul 16 21:56:16.150439 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:16.149748 2577 scope.go:117] "RemoveContainer" containerID="0fb216a341e73990938578a7e586ab70fcd9a2a89420537891248e445db0cc60" Jul 16 21:56:16.150439 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:16.149929 2577 scope.go:117] "RemoveContainer" containerID="0ccaa8e29bef6bbb701f3fba8ea7228a2445598fdc1b27d24b8c9553b97c7be6" Jul 16 21:56:16.150439 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:56:16.150214 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9_odh-ai-gateway-infra(920acc96-c6c8-4424-a663-4b994883cd58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9" podUID="920acc96-c6c8-4424-a663-4b994883cd58" Jul 16 21:56:16.166515 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:16.166496 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a885791-e3fc-4c2f-96d0-2eaec48b1ff4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9\" (UID: \"3a885791-e3fc-4c2f-96d0-2eaec48b1ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9" Jul 16 21:56:16.167077 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:16.167060 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a885791-e3fc-4c2f-96d0-2eaec48b1ff4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9\" (UID: \"3a885791-e3fc-4c2f-96d0-2eaec48b1ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9" Jul 16 21:56:16.329643 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:16.329605 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9" Jul 16 21:56:16.480628 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:16.480596 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9"] Jul 16 21:56:16.485308 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:56:16.485282 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a885791_e3fc_4c2f_96d0_2eaec48b1ff4.slice/crio-5813325e20ecfc85c95c09d28b8e35a4b94f0afb9b50efa47c99223cbfcb5be2 WatchSource:0}: Error finding container 5813325e20ecfc85c95c09d28b8e35a4b94f0afb9b50efa47c99223cbfcb5be2: Status 404 returned error can't find the container with id 5813325e20ecfc85c95c09d28b8e35a4b94f0afb9b50efa47c99223cbfcb5be2 Jul 16 21:56:17.156582 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:17.156548 2577 generic.go:356] "Generic (PLEG): container finished" podID="3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" containerID="5ab1164a2d42b99c3c0370cea4b530b14c4cb805099b6775d51254f667fd158a" exitCode=6 Jul 16 21:56:17.156968 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:17.156641 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9" event={"ID":"3a885791-e3fc-4c2f-96d0-2eaec48b1ff4","Type":"ContainerDied","Data":"5ab1164a2d42b99c3c0370cea4b530b14c4cb805099b6775d51254f667fd158a"} Jul 16 21:56:17.156968 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:17.156696 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9" event={"ID":"3a885791-e3fc-4c2f-96d0-2eaec48b1ff4","Type":"ContainerStarted","Data":"5813325e20ecfc85c95c09d28b8e35a4b94f0afb9b50efa47c99223cbfcb5be2"} Jul 16 21:56:17.156968 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:17.156817 2577 scope.go:117] "RemoveContainer" containerID="5ab1164a2d42b99c3c0370cea4b530b14c4cb805099b6775d51254f667fd158a" Jul 16 21:56:17.158540 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:17.158525 2577 scope.go:117] "RemoveContainer" containerID="0ccaa8e29bef6bbb701f3fba8ea7228a2445598fdc1b27d24b8c9553b97c7be6" Jul 16 21:56:17.158728 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:56:17.158709 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9_odh-ai-gateway-infra(920acc96-c6c8-4424-a663-4b994883cd58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9" podUID="920acc96-c6c8-4424-a663-4b994883cd58" Jul 16 21:56:18.165021 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:18.164985 2577 generic.go:356] "Generic (PLEG): container finished" podID="3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" containerID="aa2e0ccc6cfd60bcf87e1ff1deb60d26857b63e07282452d3d9b9057bf8f4302" exitCode=6 Jul 16 21:56:18.165479 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:18.165074 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9" event={"ID":"3a885791-e3fc-4c2f-96d0-2eaec48b1ff4","Type":"ContainerDied","Data":"aa2e0ccc6cfd60bcf87e1ff1deb60d26857b63e07282452d3d9b9057bf8f4302"} Jul 16 21:56:18.165479 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:18.165167 2577 scope.go:117] "RemoveContainer" containerID="5ab1164a2d42b99c3c0370cea4b530b14c4cb805099b6775d51254f667fd158a" Jul 16 21:56:18.165479 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:18.165320 2577 scope.go:117] "RemoveContainer" containerID="aa2e0ccc6cfd60bcf87e1ff1deb60d26857b63e07282452d3d9b9057bf8f4302" Jul 16 21:56:18.165653 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:56:18.165552 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9_odh-ai-gateway-infra(3a885791-e3fc-4c2f-96d0-2eaec48b1ff4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9" podUID="3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" Jul 16 21:56:19.171660 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:19.171629 2577 scope.go:117] "RemoveContainer" containerID="aa2e0ccc6cfd60bcf87e1ff1deb60d26857b63e07282452d3d9b9057bf8f4302" Jul 16 21:56:19.172109 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:56:19.171845 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9_odh-ai-gateway-infra(3a885791-e3fc-4c2f-96d0-2eaec48b1ff4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9" podUID="3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" Jul 16 21:56:27.167849 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:27.167820 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl"] Jul 16 21:56:27.171351 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:27.171329 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl" Jul 16 21:56:27.180474 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:27.180453 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl"] Jul 16 21:56:27.256116 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:27.256082 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bddd9c68-d336-46eb-bd11-c01d30aa11b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl\" (UID: \"bddd9c68-d336-46eb-bd11-c01d30aa11b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl" Jul 16 21:56:27.356844 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:27.356818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bddd9c68-d336-46eb-bd11-c01d30aa11b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl\" (UID: \"bddd9c68-d336-46eb-bd11-c01d30aa11b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl" Jul 16 21:56:27.357430 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:27.357409 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bddd9c68-d336-46eb-bd11-c01d30aa11b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl\" (UID: \"bddd9c68-d336-46eb-bd11-c01d30aa11b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl" Jul 16 21:56:27.483001 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:27.482941 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl" Jul 16 21:56:27.626773 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:27.626738 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl"] Jul 16 21:56:27.635576 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:56:27.635543 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbddd9c68_d336_46eb_bd11_c01d30aa11b9.slice/crio-0d6ade594952a37d393ede1482d58fbbcc479984fd792d0986759045ca6c7cb1 WatchSource:0}: Error finding container 0d6ade594952a37d393ede1482d58fbbcc479984fd792d0986759045ca6c7cb1: Status 404 returned error can't find the container with id 0d6ade594952a37d393ede1482d58fbbcc479984fd792d0986759045ca6c7cb1 Jul 16 21:56:28.221155 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:28.221118 2577 generic.go:356] "Generic (PLEG): container finished" podID="bddd9c68-d336-46eb-bd11-c01d30aa11b9" containerID="4b3aeda07b854b73a8f19b972a8aa84c9989bea54c55d58ee70c2275134ab0f6" exitCode=6 Jul 16 21:56:28.221574 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:28.221172 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl" event={"ID":"bddd9c68-d336-46eb-bd11-c01d30aa11b9","Type":"ContainerDied","Data":"4b3aeda07b854b73a8f19b972a8aa84c9989bea54c55d58ee70c2275134ab0f6"} Jul 16 21:56:28.221574 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:28.221259 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl" event={"ID":"bddd9c68-d336-46eb-bd11-c01d30aa11b9","Type":"ContainerStarted","Data":"0d6ade594952a37d393ede1482d58fbbcc479984fd792d0986759045ca6c7cb1"} Jul 16 21:56:28.221574 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:28.221492 2577 scope.go:117] "RemoveContainer" containerID="4b3aeda07b854b73a8f19b972a8aa84c9989bea54c55d58ee70c2275134ab0f6" Jul 16 21:56:29.227909 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:29.227832 2577 generic.go:356] "Generic (PLEG): container finished" podID="bddd9c68-d336-46eb-bd11-c01d30aa11b9" containerID="cc532ba80176371b1ca0841e7a4ff65d3f13208a27f0f107f5bebc5c51dab0f9" exitCode=6 Jul 16 21:56:29.227909 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:29.227866 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl" event={"ID":"bddd9c68-d336-46eb-bd11-c01d30aa11b9","Type":"ContainerDied","Data":"cc532ba80176371b1ca0841e7a4ff65d3f13208a27f0f107f5bebc5c51dab0f9"} Jul 16 21:56:29.227909 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:29.227907 2577 scope.go:117] "RemoveContainer" containerID="4b3aeda07b854b73a8f19b972a8aa84c9989bea54c55d58ee70c2275134ab0f6" Jul 16 21:56:29.228447 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:29.228106 2577 scope.go:117] "RemoveContainer" containerID="cc532ba80176371b1ca0841e7a4ff65d3f13208a27f0f107f5bebc5c51dab0f9" Jul 16 21:56:29.228447 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:56:29.228326 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl_odh-ai-gateway-infra(bddd9c68-d336-46eb-bd11-c01d30aa11b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl" podUID="bddd9c68-d336-46eb-bd11-c01d30aa11b9" Jul 16 21:56:30.234086 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:30.234059 2577 scope.go:117] "RemoveContainer" containerID="cc532ba80176371b1ca0841e7a4ff65d3f13208a27f0f107f5bebc5c51dab0f9" Jul 16 21:56:30.234459 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:56:30.234248 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl_odh-ai-gateway-infra(bddd9c68-d336-46eb-bd11-c01d30aa11b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl" podUID="bddd9c68-d336-46eb-bd11-c01d30aa11b9" Jul 16 21:56:30.450376 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:30.450350 2577 scope.go:117] "RemoveContainer" containerID="0ccaa8e29bef6bbb701f3fba8ea7228a2445598fdc1b27d24b8c9553b97c7be6" Jul 16 21:56:31.240412 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:31.240385 2577 generic.go:356] "Generic (PLEG): container finished" podID="920acc96-c6c8-4424-a663-4b994883cd58" containerID="6df5b5c7af550d51fda7d62192f9826511761918502e3279f101fb483720f38a" exitCode=6 Jul 16 21:56:31.240800 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:31.240440 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9" event={"ID":"920acc96-c6c8-4424-a663-4b994883cd58","Type":"ContainerDied","Data":"6df5b5c7af550d51fda7d62192f9826511761918502e3279f101fb483720f38a"} Jul 16 21:56:31.240800 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:31.240504 2577 scope.go:117] "RemoveContainer" containerID="0ccaa8e29bef6bbb701f3fba8ea7228a2445598fdc1b27d24b8c9553b97c7be6" Jul 16 21:56:31.240800 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:31.240760 2577 scope.go:117] "RemoveContainer" containerID="6df5b5c7af550d51fda7d62192f9826511761918502e3279f101fb483720f38a" Jul 16 21:56:31.240953 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:56:31.240933 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9_odh-ai-gateway-infra(920acc96-c6c8-4424-a663-4b994883cd58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9" podUID="920acc96-c6c8-4424-a663-4b994883cd58" Jul 16 21:56:32.270659 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:32.270625 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9"] Jul 16 21:56:32.411409 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:32.411387 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9" Jul 16 21:56:32.498487 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:32.498459 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920acc96-c6c8-4424-a663-4b994883cd58-maas-api-service-ca\") pod \"920acc96-c6c8-4424-a663-4b994883cd58\" (UID: \"920acc96-c6c8-4424-a663-4b994883cd58\") " Jul 16 21:56:32.498791 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:32.498770 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920acc96-c6c8-4424-a663-4b994883cd58-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "920acc96-c6c8-4424-a663-4b994883cd58" (UID: "920acc96-c6c8-4424-a663-4b994883cd58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:56:32.599455 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:32.599382 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920acc96-c6c8-4424-a663-4b994883cd58-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:56:33.253097 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:33.253069 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9" Jul 16 21:56:33.253293 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:33.253069 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9" event={"ID":"920acc96-c6c8-4424-a663-4b994883cd58","Type":"ContainerDied","Data":"57fe2e09776c755400553b15d08e81afe128b60e113bb8cfc63bf2eab447cf4e"} Jul 16 21:56:33.253293 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:33.253224 2577 scope.go:117] "RemoveContainer" containerID="6df5b5c7af550d51fda7d62192f9826511761918502e3279f101fb483720f38a" Jul 16 21:56:33.279453 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:33.279427 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9"] Jul 16 21:56:33.284743 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:33.284724 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-jfbf9"] Jul 16 21:56:34.451159 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:34.451119 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="920acc96-c6c8-4424-a663-4b994883cd58" path="/var/lib/kubelet/pods/920acc96-c6c8-4424-a663-4b994883cd58/volumes" Jul 16 21:56:34.451867 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:34.451848 2577 scope.go:117] "RemoveContainer" containerID="aa2e0ccc6cfd60bcf87e1ff1deb60d26857b63e07282452d3d9b9057bf8f4302" Jul 16 21:56:35.266625 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.266595 2577 generic.go:356] "Generic (PLEG): container finished" podID="3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" containerID="57abfaf60854f5d0d63d8abcd28034ec5982d16834bfc3ddd4ed935e84097cca" exitCode=6 Jul 16 21:56:35.266805 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.266633 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9" event={"ID":"3a885791-e3fc-4c2f-96d0-2eaec48b1ff4","Type":"ContainerDied","Data":"57abfaf60854f5d0d63d8abcd28034ec5982d16834bfc3ddd4ed935e84097cca"} Jul 16 21:56:35.266805 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.266679 2577 scope.go:117] "RemoveContainer" containerID="aa2e0ccc6cfd60bcf87e1ff1deb60d26857b63e07282452d3d9b9057bf8f4302" Jul 16 21:56:35.267040 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.267021 2577 scope.go:117] "RemoveContainer" containerID="57abfaf60854f5d0d63d8abcd28034ec5982d16834bfc3ddd4ed935e84097cca" Jul 16 21:56:35.267297 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:56:35.267277 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9_odh-ai-gateway-infra(3a885791-e3fc-4c2f-96d0-2eaec48b1ff4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9" podUID="3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" Jul 16 21:56:35.417957 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.417928 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz"] Jul 16 21:56:35.418289 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.418276 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920acc96-c6c8-4424-a663-4b994883cd58" containerName="revoke-keys" Jul 16 21:56:35.418356 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.418291 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="920acc96-c6c8-4424-a663-4b994883cd58" containerName="revoke-keys" Jul 16 21:56:35.418356 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.418303 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920acc96-c6c8-4424-a663-4b994883cd58" containerName="revoke-keys" Jul 16 21:56:35.418356 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.418309 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="920acc96-c6c8-4424-a663-4b994883cd58" containerName="revoke-keys" Jul 16 21:56:35.418458 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.418376 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="920acc96-c6c8-4424-a663-4b994883cd58" containerName="revoke-keys" Jul 16 21:56:35.418458 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.418386 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="920acc96-c6c8-4424-a663-4b994883cd58" containerName="revoke-keys" Jul 16 21:56:35.422755 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.422741 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" Jul 16 21:56:35.430678 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.430655 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz"] Jul 16 21:56:35.528001 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.527967 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e367f8-d0bc-440e-a94f-cdbb5256c502-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz\" (UID: \"04e367f8-d0bc-440e-a94f-cdbb5256c502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" Jul 16 21:56:35.629229 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.629175 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e367f8-d0bc-440e-a94f-cdbb5256c502-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz\" (UID: \"04e367f8-d0bc-440e-a94f-cdbb5256c502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" Jul 16 21:56:35.629784 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.629766 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e367f8-d0bc-440e-a94f-cdbb5256c502-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz\" (UID: \"04e367f8-d0bc-440e-a94f-cdbb5256c502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" Jul 16 21:56:35.734879 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.734786 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" Jul 16 21:56:35.867200 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:35.867160 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz"] Jul 16 21:56:35.871966 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:56:35.871928 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e367f8_d0bc_440e_a94f_cdbb5256c502.slice/crio-a10e7a9e77ab5e27a68d5318e45f73b927b9c4838fd83a1a57323b8c694c0150 WatchSource:0}: Error finding container a10e7a9e77ab5e27a68d5318e45f73b927b9c4838fd83a1a57323b8c694c0150: Status 404 returned error can't find the container with id a10e7a9e77ab5e27a68d5318e45f73b927b9c4838fd83a1a57323b8c694c0150 Jul 16 21:56:36.274586 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:36.274500 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" event={"ID":"04e367f8-d0bc-440e-a94f-cdbb5256c502","Type":"ContainerStarted","Data":"9707fb8f8c7e7a44c1719cc9d2c7347ebbabab3fed621f7e700b4f163f946b19"} Jul 16 21:56:36.280617 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:36.274857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" event={"ID":"04e367f8-d0bc-440e-a94f-cdbb5256c502","Type":"ContainerStarted","Data":"a10e7a9e77ab5e27a68d5318e45f73b927b9c4838fd83a1a57323b8c694c0150"} Jul 16 21:56:36.295598 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:36.295553 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" podStartSLOduration=1.295539848 podStartE2EDuration="1.295539848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:36.294456645 +0000 UTC m=+2812.469564858" watchObservedRunningTime="2026-07-16 21:56:36.295539848 +0000 UTC m=+2812.470648040" Jul 16 21:56:36.299529 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:36.299502 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9"] Jul 16 21:56:36.434975 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:36.434944 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9" Jul 16 21:56:36.538108 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:36.538040 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a885791-e3fc-4c2f-96d0-2eaec48b1ff4-maas-api-service-ca\") pod \"3a885791-e3fc-4c2f-96d0-2eaec48b1ff4\" (UID: \"3a885791-e3fc-4c2f-96d0-2eaec48b1ff4\") " Jul 16 21:56:36.538499 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:36.538413 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a885791-e3fc-4c2f-96d0-2eaec48b1ff4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" (UID: "3a885791-e3fc-4c2f-96d0-2eaec48b1ff4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:56:36.639484 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:36.639458 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a885791-e3fc-4c2f-96d0-2eaec48b1ff4-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:56:37.281322 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:37.281282 2577 generic.go:356] "Generic (PLEG): container finished" podID="04e367f8-d0bc-440e-a94f-cdbb5256c502" containerID="9707fb8f8c7e7a44c1719cc9d2c7347ebbabab3fed621f7e700b4f163f946b19" exitCode=6 Jul 16 21:56:37.281521 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:37.281371 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" event={"ID":"04e367f8-d0bc-440e-a94f-cdbb5256c502","Type":"ContainerDied","Data":"9707fb8f8c7e7a44c1719cc9d2c7347ebbabab3fed621f7e700b4f163f946b19"} Jul 16 21:56:37.281600 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:37.281536 2577 scope.go:117] "RemoveContainer" containerID="9707fb8f8c7e7a44c1719cc9d2c7347ebbabab3fed621f7e700b4f163f946b19" Jul 16 21:56:37.282653 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:37.282630 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9" event={"ID":"3a885791-e3fc-4c2f-96d0-2eaec48b1ff4","Type":"ContainerDied","Data":"5813325e20ecfc85c95c09d28b8e35a4b94f0afb9b50efa47c99223cbfcb5be2"} Jul 16 21:56:37.282763 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:37.282660 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9" Jul 16 21:56:37.282763 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:37.282688 2577 scope.go:117] "RemoveContainer" containerID="57abfaf60854f5d0d63d8abcd28034ec5982d16834bfc3ddd4ed935e84097cca" Jul 16 21:56:37.314120 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:37.314096 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9"] Jul 16 21:56:37.320467 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:37.320447 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-fv8w9"] Jul 16 21:56:38.290143 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:38.290057 2577 generic.go:356] "Generic (PLEG): container finished" podID="04e367f8-d0bc-440e-a94f-cdbb5256c502" containerID="b4671d675d7835560dc4753be8b46a8fefea71c59be30c24a48aca27b1d74266" exitCode=6 Jul 16 21:56:38.290624 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:38.290146 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" event={"ID":"04e367f8-d0bc-440e-a94f-cdbb5256c502","Type":"ContainerDied","Data":"b4671d675d7835560dc4753be8b46a8fefea71c59be30c24a48aca27b1d74266"} Jul 16 21:56:38.290624 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:38.290231 2577 scope.go:117] "RemoveContainer" containerID="9707fb8f8c7e7a44c1719cc9d2c7347ebbabab3fed621f7e700b4f163f946b19" Jul 16 21:56:38.290624 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:38.290376 2577 scope.go:117] "RemoveContainer" containerID="b4671d675d7835560dc4753be8b46a8fefea71c59be30c24a48aca27b1d74266" Jul 16 21:56:38.290802 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:56:38.290615 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz_odh-ai-gateway-infra(04e367f8-d0bc-440e-a94f-cdbb5256c502)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" podUID="04e367f8-d0bc-440e-a94f-cdbb5256c502" Jul 16 21:56:38.450288 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:38.450258 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" path="/var/lib/kubelet/pods/3a885791-e3fc-4c2f-96d0-2eaec48b1ff4/volumes" Jul 16 21:56:39.298036 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:39.298008 2577 scope.go:117] "RemoveContainer" containerID="b4671d675d7835560dc4753be8b46a8fefea71c59be30c24a48aca27b1d74266" Jul 16 21:56:39.298434 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:56:39.298171 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz_odh-ai-gateway-infra(04e367f8-d0bc-440e-a94f-cdbb5256c502)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" podUID="04e367f8-d0bc-440e-a94f-cdbb5256c502" Jul 16 21:56:42.445260 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:42.445230 2577 scope.go:117] "RemoveContainer" containerID="cc532ba80176371b1ca0841e7a4ff65d3f13208a27f0f107f5bebc5c51dab0f9" Jul 16 21:56:43.319657 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:43.319627 2577 generic.go:356] "Generic (PLEG): container finished" podID="bddd9c68-d336-46eb-bd11-c01d30aa11b9" containerID="0cb53a4b79d7e6165c1db61f1c15fa9de5f8d9de88b5571e2895166c5fa04341" exitCode=6 Jul 16 21:56:43.319802 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:43.319682 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl" event={"ID":"bddd9c68-d336-46eb-bd11-c01d30aa11b9","Type":"ContainerDied","Data":"0cb53a4b79d7e6165c1db61f1c15fa9de5f8d9de88b5571e2895166c5fa04341"} Jul 16 21:56:43.319802 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:43.319727 2577 scope.go:117] "RemoveContainer" containerID="cc532ba80176371b1ca0841e7a4ff65d3f13208a27f0f107f5bebc5c51dab0f9" Jul 16 21:56:43.320038 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:43.320022 2577 scope.go:117] "RemoveContainer" containerID="0cb53a4b79d7e6165c1db61f1c15fa9de5f8d9de88b5571e2895166c5fa04341" Jul 16 21:56:43.320273 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:56:43.320253 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl_odh-ai-gateway-infra(bddd9c68-d336-46eb-bd11-c01d30aa11b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl" podUID="bddd9c68-d336-46eb-bd11-c01d30aa11b9" Jul 16 21:56:44.351126 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:44.351093 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl"] Jul 16 21:56:44.487135 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:44.487112 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl" Jul 16 21:56:44.601816 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:44.601743 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bddd9c68-d336-46eb-bd11-c01d30aa11b9-maas-api-service-ca\") pod \"bddd9c68-d336-46eb-bd11-c01d30aa11b9\" (UID: \"bddd9c68-d336-46eb-bd11-c01d30aa11b9\") " Jul 16 21:56:44.602088 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:44.602065 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bddd9c68-d336-46eb-bd11-c01d30aa11b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bddd9c68-d336-46eb-bd11-c01d30aa11b9" (UID: "bddd9c68-d336-46eb-bd11-c01d30aa11b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:56:44.702507 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:44.702480 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bddd9c68-d336-46eb-bd11-c01d30aa11b9-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:56:45.333081 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:45.333052 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl" Jul 16 21:56:45.333081 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:45.333076 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl" event={"ID":"bddd9c68-d336-46eb-bd11-c01d30aa11b9","Type":"ContainerDied","Data":"0d6ade594952a37d393ede1482d58fbbcc479984fd792d0986759045ca6c7cb1"} Jul 16 21:56:45.333367 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:45.333134 2577 scope.go:117] "RemoveContainer" containerID="0cb53a4b79d7e6165c1db61f1c15fa9de5f8d9de88b5571e2895166c5fa04341" Jul 16 21:56:45.359909 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:45.359798 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl"] Jul 16 21:56:45.366096 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:45.366072 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-q2cfl"] Jul 16 21:56:46.451197 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:46.451154 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bddd9c68-d336-46eb-bd11-c01d30aa11b9" path="/var/lib/kubelet/pods/bddd9c68-d336-46eb-bd11-c01d30aa11b9/volumes" Jul 16 21:56:50.445261 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:50.445235 2577 scope.go:117] "RemoveContainer" containerID="b4671d675d7835560dc4753be8b46a8fefea71c59be30c24a48aca27b1d74266" Jul 16 21:56:51.365476 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:51.365392 2577 generic.go:356] "Generic (PLEG): container finished" podID="04e367f8-d0bc-440e-a94f-cdbb5256c502" containerID="1a85241cc3d5b35b18780025427033d6dbe1208e8a5e099fc395ad6e75f7d57a" exitCode=6 Jul 16 21:56:51.365476 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:51.365456 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" event={"ID":"04e367f8-d0bc-440e-a94f-cdbb5256c502","Type":"ContainerDied","Data":"1a85241cc3d5b35b18780025427033d6dbe1208e8a5e099fc395ad6e75f7d57a"} Jul 16 21:56:51.365652 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:51.365516 2577 scope.go:117] "RemoveContainer" containerID="b4671d675d7835560dc4753be8b46a8fefea71c59be30c24a48aca27b1d74266" Jul 16 21:56:51.365794 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:51.365778 2577 scope.go:117] "RemoveContainer" containerID="1a85241cc3d5b35b18780025427033d6dbe1208e8a5e099fc395ad6e75f7d57a" Jul 16 21:56:51.366015 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:56:51.365988 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz_odh-ai-gateway-infra(04e367f8-d0bc-440e-a94f-cdbb5256c502)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" podUID="04e367f8-d0bc-440e-a94f-cdbb5256c502" Jul 16 21:56:52.395316 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:52.395285 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz"] Jul 16 21:56:52.525308 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:52.525283 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" Jul 16 21:56:52.661565 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:52.661482 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e367f8-d0bc-440e-a94f-cdbb5256c502-maas-api-service-ca\") pod \"04e367f8-d0bc-440e-a94f-cdbb5256c502\" (UID: \"04e367f8-d0bc-440e-a94f-cdbb5256c502\") " Jul 16 21:56:52.661884 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:52.661851 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e367f8-d0bc-440e-a94f-cdbb5256c502-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04e367f8-d0bc-440e-a94f-cdbb5256c502" (UID: "04e367f8-d0bc-440e-a94f-cdbb5256c502"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:56:52.762697 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:52.762669 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e367f8-d0bc-440e-a94f-cdbb5256c502-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:56:53.385157 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:53.385122 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" event={"ID":"04e367f8-d0bc-440e-a94f-cdbb5256c502","Type":"ContainerDied","Data":"a10e7a9e77ab5e27a68d5318e45f73b927b9c4838fd83a1a57323b8c694c0150"} Jul 16 21:56:53.385157 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:53.385147 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz" Jul 16 21:56:53.385428 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:53.385207 2577 scope.go:117] "RemoveContainer" containerID="1a85241cc3d5b35b18780025427033d6dbe1208e8a5e099fc395ad6e75f7d57a" Jul 16 21:56:53.431842 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:53.431812 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz"] Jul 16 21:56:53.442317 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:53.442298 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rc2hz"] Jul 16 21:56:54.450507 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:56:54.450477 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04e367f8-d0bc-440e-a94f-cdbb5256c502" path="/var/lib/kubelet/pods/04e367f8-d0bc-440e-a94f-cdbb5256c502/volumes" Jul 16 21:57:03.805510 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805465 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd"] Jul 16 21:57:03.805895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805805 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" containerName="revoke-keys" Jul 16 21:57:03.805895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805816 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" containerName="revoke-keys" Jul 16 21:57:03.805895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805824 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" containerName="revoke-keys" Jul 16 21:57:03.805895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805829 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" containerName="revoke-keys" Jul 16 21:57:03.805895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805838 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e367f8-d0bc-440e-a94f-cdbb5256c502" containerName="revoke-keys" Jul 16 21:57:03.805895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805845 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e367f8-d0bc-440e-a94f-cdbb5256c502" containerName="revoke-keys" Jul 16 21:57:03.805895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805853 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e367f8-d0bc-440e-a94f-cdbb5256c502" containerName="revoke-keys" Jul 16 21:57:03.805895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805859 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e367f8-d0bc-440e-a94f-cdbb5256c502" containerName="revoke-keys" Jul 16 21:57:03.805895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805864 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e367f8-d0bc-440e-a94f-cdbb5256c502" containerName="revoke-keys" Jul 16 21:57:03.805895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805869 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e367f8-d0bc-440e-a94f-cdbb5256c502" containerName="revoke-keys" Jul 16 21:57:03.805895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805876 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920acc96-c6c8-4424-a663-4b994883cd58" containerName="revoke-keys" Jul 16 21:57:03.805895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805880 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="920acc96-c6c8-4424-a663-4b994883cd58" containerName="revoke-keys" Jul 16 21:57:03.805895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805887 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bddd9c68-d336-46eb-bd11-c01d30aa11b9" containerName="revoke-keys" Jul 16 21:57:03.805895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805892 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bddd9c68-d336-46eb-bd11-c01d30aa11b9" containerName="revoke-keys" Jul 16 21:57:03.805895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805901 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bddd9c68-d336-46eb-bd11-c01d30aa11b9" containerName="revoke-keys" Jul 16 21:57:03.806385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805906 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bddd9c68-d336-46eb-bd11-c01d30aa11b9" containerName="revoke-keys" Jul 16 21:57:03.806385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805915 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bddd9c68-d336-46eb-bd11-c01d30aa11b9" containerName="revoke-keys" Jul 16 21:57:03.806385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805919 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bddd9c68-d336-46eb-bd11-c01d30aa11b9" containerName="revoke-keys" Jul 16 21:57:03.806385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805970 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e367f8-d0bc-440e-a94f-cdbb5256c502" containerName="revoke-keys" Jul 16 21:57:03.806385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805977 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" containerName="revoke-keys" Jul 16 21:57:03.806385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805982 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bddd9c68-d336-46eb-bd11-c01d30aa11b9" containerName="revoke-keys" Jul 16 21:57:03.806385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805989 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e367f8-d0bc-440e-a94f-cdbb5256c502" containerName="revoke-keys" Jul 16 21:57:03.806385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805994 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" containerName="revoke-keys" Jul 16 21:57:03.806385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.805999 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bddd9c68-d336-46eb-bd11-c01d30aa11b9" containerName="revoke-keys" Jul 16 21:57:03.806385 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.806004 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="920acc96-c6c8-4424-a663-4b994883cd58" containerName="revoke-keys" Jul 16 21:57:03.809997 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.809981 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd" Jul 16 21:57:03.814736 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.814716 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:57:03.814951 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.814936 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:57:03.839620 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.839594 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd"] Jul 16 21:57:03.855796 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.855763 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62808ce-0937-4973-91f4-80e161074555-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd\" (UID: \"d62808ce-0937-4973-91f4-80e161074555\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd" Jul 16 21:57:03.956965 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.956928 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62808ce-0937-4973-91f4-80e161074555-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd\" (UID: \"d62808ce-0937-4973-91f4-80e161074555\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd" Jul 16 21:57:03.957557 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:03.957538 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62808ce-0937-4973-91f4-80e161074555-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd\" (UID: \"d62808ce-0937-4973-91f4-80e161074555\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd" Jul 16 21:57:04.120365 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:04.120285 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd" Jul 16 21:57:04.269146 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:57:04.269116 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62808ce_0937_4973_91f4_80e161074555.slice/crio-367a8c02542ad075f96e3422d914030071d3331759b6a31b4d9c48e190deb710 WatchSource:0}: Error finding container 367a8c02542ad075f96e3422d914030071d3331759b6a31b4d9c48e190deb710: Status 404 returned error can't find the container with id 367a8c02542ad075f96e3422d914030071d3331759b6a31b4d9c48e190deb710 Jul 16 21:57:04.270497 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:04.270476 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd"] Jul 16 21:57:04.444240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:04.444203 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd" event={"ID":"d62808ce-0937-4973-91f4-80e161074555","Type":"ContainerStarted","Data":"367a8c02542ad075f96e3422d914030071d3331759b6a31b4d9c48e190deb710"} Jul 16 21:57:05.450325 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:05.450291 2577 generic.go:356] "Generic (PLEG): container finished" podID="d62808ce-0937-4973-91f4-80e161074555" containerID="0d8dcfb78651adea6a3915d21dddab1ddf97f24199f7e6a0e7df4b26a08122f6" exitCode=6 Jul 16 21:57:05.450730 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:05.450384 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd" event={"ID":"d62808ce-0937-4973-91f4-80e161074555","Type":"ContainerDied","Data":"0d8dcfb78651adea6a3915d21dddab1ddf97f24199f7e6a0e7df4b26a08122f6"} Jul 16 21:57:05.450730 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:05.450525 2577 scope.go:117] "RemoveContainer" containerID="0d8dcfb78651adea6a3915d21dddab1ddf97f24199f7e6a0e7df4b26a08122f6" Jul 16 21:57:06.456254 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.456158 2577 generic.go:356] "Generic (PLEG): container finished" podID="d62808ce-0937-4973-91f4-80e161074555" containerID="0e5cd0dca5626f9d40cc36e021bc3b6924cf944cc5d448194eff01fd6e17dc48" exitCode=6 Jul 16 21:57:06.456254 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.456219 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd" event={"ID":"d62808ce-0937-4973-91f4-80e161074555","Type":"ContainerDied","Data":"0e5cd0dca5626f9d40cc36e021bc3b6924cf944cc5d448194eff01fd6e17dc48"} Jul 16 21:57:06.456764 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.456275 2577 scope.go:117] "RemoveContainer" containerID="0d8dcfb78651adea6a3915d21dddab1ddf97f24199f7e6a0e7df4b26a08122f6" Jul 16 21:57:06.456764 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.456464 2577 scope.go:117] "RemoveContainer" containerID="0e5cd0dca5626f9d40cc36e021bc3b6924cf944cc5d448194eff01fd6e17dc48" Jul 16 21:57:06.456764 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:57:06.456687 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd_odh-ai-gateway-infra(d62808ce-0937-4973-91f4-80e161074555)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd" podUID="d62808ce-0937-4973-91f4-80e161074555" Jul 16 21:57:06.637712 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.637683 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5"] Jul 16 21:57:06.638038 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.638027 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" containerName="revoke-keys" Jul 16 21:57:06.638130 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.638040 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" containerName="revoke-keys" Jul 16 21:57:06.638130 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.638100 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a885791-e3fc-4c2f-96d0-2eaec48b1ff4" containerName="revoke-keys" Jul 16 21:57:06.638130 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.638110 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e367f8-d0bc-440e-a94f-cdbb5256c502" containerName="revoke-keys" Jul 16 21:57:06.638130 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.638116 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bddd9c68-d336-46eb-bd11-c01d30aa11b9" containerName="revoke-keys" Jul 16 21:57:06.642300 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.642285 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" Jul 16 21:57:06.645693 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.645673 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xcg5c\"" Jul 16 21:57:06.645862 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.645847 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 21:57:06.646611 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.646593 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f53605-serving-cert\"" Jul 16 21:57:06.657480 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.657460 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5"] Jul 16 21:57:06.676608 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.676580 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4gcn\" (UniqueName: \"kubernetes.io/projected/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-kube-api-access-l4gcn\") pod \"maas-api-e2e-inf-a-f53605-68595969cc-mdfk5\" (UID: \"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" Jul 16 21:57:06.676757 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.676623 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f53605-68595969cc-mdfk5\" (UID: \"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" Jul 16 21:57:06.676757 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.676733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-maas-api-tls\") pod \"maas-api-e2e-inf-a-f53605-68595969cc-mdfk5\" (UID: \"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" Jul 16 21:57:06.777757 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.777674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4gcn\" (UniqueName: \"kubernetes.io/projected/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-kube-api-access-l4gcn\") pod \"maas-api-e2e-inf-a-f53605-68595969cc-mdfk5\" (UID: \"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" Jul 16 21:57:06.777757 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.777720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f53605-68595969cc-mdfk5\" (UID: \"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" Jul 16 21:57:06.777963 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.777779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-maas-api-tls\") pod \"maas-api-e2e-inf-a-f53605-68595969cc-mdfk5\" (UID: \"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" Jul 16 21:57:06.777963 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:57:06.777905 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-serving-cert: secret "maas-api-e2e-inf-a-f53605-serving-cert" not found Jul 16 21:57:06.778050 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:57:06.777969 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-maas-api-tls podName:cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc nodeName:}" failed. No retries permitted until 2026-07-16 21:57:07.277949808 +0000 UTC m=+2843.453057991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-maas-api-tls") pod "maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" (UID: "cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc") : secret "maas-api-e2e-inf-a-f53605-serving-cert" not found Jul 16 21:57:06.778486 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.778468 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f53605-68595969cc-mdfk5\" (UID: \"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" Jul 16 21:57:06.789029 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:06.789007 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4gcn\" (UniqueName: \"kubernetes.io/projected/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-kube-api-access-l4gcn\") pod \"maas-api-e2e-inf-a-f53605-68595969cc-mdfk5\" (UID: \"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" Jul 16 21:57:07.281255 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:07.281216 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-maas-api-tls\") pod \"maas-api-e2e-inf-a-f53605-68595969cc-mdfk5\" (UID: \"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" Jul 16 21:57:07.283551 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:07.283534 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-maas-api-tls\") pod \"maas-api-e2e-inf-a-f53605-68595969cc-mdfk5\" (UID: \"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" Jul 16 21:57:07.462769 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:07.462740 2577 scope.go:117] "RemoveContainer" containerID="0e5cd0dca5626f9d40cc36e021bc3b6924cf944cc5d448194eff01fd6e17dc48" Jul 16 21:57:07.463120 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:57:07.462922 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd_odh-ai-gateway-infra(d62808ce-0937-4973-91f4-80e161074555)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd" podUID="d62808ce-0937-4973-91f4-80e161074555" Jul 16 21:57:07.552483 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:07.552403 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" Jul 16 21:57:07.710423 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:07.710399 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5"] Jul 16 21:57:07.713086 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:57:07.713063 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8ec15a_d6c8_44f6_822c_c1893eb1ebcc.slice/crio-4bf657b8405958d2fab5c360b33e58cdf0689b1796dae1e189a84821a48c9b4d WatchSource:0}: Error finding container 4bf657b8405958d2fab5c360b33e58cdf0689b1796dae1e189a84821a48c9b4d: Status 404 returned error can't find the container with id 4bf657b8405958d2fab5c360b33e58cdf0689b1796dae1e189a84821a48c9b4d Jul 16 21:57:08.468520 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:08.468429 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" event={"ID":"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc","Type":"ContainerStarted","Data":"2cf1bbad3abee82b00d39dd3dae83886f2b51ee1bcc67ddc4ad085b75fee9d00"} Jul 16 21:57:08.468520 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:08.468477 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" event={"ID":"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc","Type":"ContainerStarted","Data":"4bf657b8405958d2fab5c360b33e58cdf0689b1796dae1e189a84821a48c9b4d"} Jul 16 21:57:08.468933 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:08.468539 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" Jul 16 21:57:08.488846 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:08.488783 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" podStartSLOduration=2.020653157 podStartE2EDuration="2.488764333s" totalImagesPullingTime="468.111176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:08.486953668 +0000 UTC m=+2844.662061872" watchObservedRunningTime="2026-07-16 21:57:08.488764333 +0000 UTC m=+2844.663872526" Jul 16 21:57:11.654249 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.654214 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf"] Jul 16 21:57:11.657826 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.657810 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.670485 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.670467 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6e4744-openshift-default-dockercfg-lrr4k\"" Jul 16 21:57:11.682876 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.682852 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf"] Jul 16 21:57:11.714269 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.714247 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-token\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.714375 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.714282 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvpdw\" (UniqueName: \"kubernetes.io/projected/a20bca18-8725-49cd-8471-ff4a2f04aa8b-kube-api-access-hvpdw\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.714375 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.714304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-envoy\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.714375 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.714322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-data\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.714375 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.714342 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-podinfo\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.714524 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.714380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-workload-certs\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.714524 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.714434 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-workload-socket\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.714524 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.714463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istiod-ca-cert\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.714524 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.714483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-credential-socket\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.815607 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.815559 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-workload-socket\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.815780 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.815623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istiod-ca-cert\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.815780 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.815660 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-credential-socket\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.815780 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.815703 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-token\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.815780 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.815752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvpdw\" (UniqueName: \"kubernetes.io/projected/a20bca18-8725-49cd-8471-ff4a2f04aa8b-kube-api-access-hvpdw\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.815956 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.815786 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-envoy\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.816013 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.815947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-data\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.816013 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.816002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-podinfo\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.816125 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.816035 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-workload-socket\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.816125 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.816048 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-workload-certs\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.816125 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.816097 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-credential-socket\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.816326 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.816272 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-data\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.816391 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.816375 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-workload-certs\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.816469 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.816449 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istiod-ca-cert\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.817923 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.817905 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-envoy\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.818468 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.818443 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-podinfo\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.826593 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.826568 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-token\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.826701 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.826680 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvpdw\" (UniqueName: \"kubernetes.io/projected/a20bca18-8725-49cd-8471-ff4a2f04aa8b-kube-api-access-hvpdw\") pod \"e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:11.967907 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:11.967828 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:12.144413 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:12.144386 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf"] Jul 16 21:57:12.145677 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:57:12.145652 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20bca18_8725_49cd_8471_ff4a2f04aa8b.slice/crio-f39ea6a4956c811da3968c08ad1958b5c29397cfd050a6c9bf9dae56e158db33 WatchSource:0}: Error finding container f39ea6a4956c811da3968c08ad1958b5c29397cfd050a6c9bf9dae56e158db33: Status 404 returned error can't find the container with id f39ea6a4956c811da3968c08ad1958b5c29397cfd050a6c9bf9dae56e158db33 Jul 16 21:57:12.147862 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:12.147834 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:57:12.147922 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:12.147895 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:57:12.147962 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:12.147922 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:57:12.490103 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:12.490067 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" event={"ID":"a20bca18-8725-49cd-8471-ff4a2f04aa8b","Type":"ContainerStarted","Data":"5f92ec3cf32418a4b59c0c62e87e8b01c39a8949d670de2ca047608f49103cad"} Jul 16 21:57:12.490284 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:12.490118 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" event={"ID":"a20bca18-8725-49cd-8471-ff4a2f04aa8b","Type":"ContainerStarted","Data":"f39ea6a4956c811da3968c08ad1958b5c29397cfd050a6c9bf9dae56e158db33"} Jul 16 21:57:12.521636 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:12.521591 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" podStartSLOduration=1.521575854 podStartE2EDuration="1.521575854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:12.519447079 +0000 UTC m=+2848.694555270" watchObservedRunningTime="2026-07-16 21:57:12.521575854 +0000 UTC m=+2848.696684046" Jul 16 21:57:12.968237 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:12.968177 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:13.973143 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:13.973116 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:14.479395 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:14.479368 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" Jul 16 21:57:14.500346 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:14.500319 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:14.501391 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:14.501370 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:14.947518 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:14.947481 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht"] Jul 16 21:57:14.951009 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:14.950992 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" Jul 16 21:57:14.953800 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:14.953779 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6e4744-serving-cert\"" Jul 16 21:57:14.964926 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:14.964904 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht"] Jul 16 21:57:15.047522 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:15.047492 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6826aa50-1e07-4ea2-be24-e951eeb56e21-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht\" (UID: \"6826aa50-1e07-4ea2-be24-e951eeb56e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" Jul 16 21:57:15.047895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:15.047537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcbmt\" (UniqueName: \"kubernetes.io/projected/6826aa50-1e07-4ea2-be24-e951eeb56e21-kube-api-access-dcbmt\") pod \"maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht\" (UID: \"6826aa50-1e07-4ea2-be24-e951eeb56e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" Jul 16 21:57:15.047895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:15.047617 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6826aa50-1e07-4ea2-be24-e951eeb56e21-maas-api-tls\") pod \"maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht\" (UID: \"6826aa50-1e07-4ea2-be24-e951eeb56e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" Jul 16 21:57:15.148060 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:15.148026 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6826aa50-1e07-4ea2-be24-e951eeb56e21-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht\" (UID: \"6826aa50-1e07-4ea2-be24-e951eeb56e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" Jul 16 21:57:15.148244 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:15.148089 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcbmt\" (UniqueName: \"kubernetes.io/projected/6826aa50-1e07-4ea2-be24-e951eeb56e21-kube-api-access-dcbmt\") pod \"maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht\" (UID: \"6826aa50-1e07-4ea2-be24-e951eeb56e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" Jul 16 21:57:15.148244 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:15.148164 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6826aa50-1e07-4ea2-be24-e951eeb56e21-maas-api-tls\") pod \"maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht\" (UID: \"6826aa50-1e07-4ea2-be24-e951eeb56e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" Jul 16 21:57:15.148781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:15.148757 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6826aa50-1e07-4ea2-be24-e951eeb56e21-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht\" (UID: \"6826aa50-1e07-4ea2-be24-e951eeb56e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" Jul 16 21:57:15.150850 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:15.150830 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6826aa50-1e07-4ea2-be24-e951eeb56e21-maas-api-tls\") pod \"maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht\" (UID: \"6826aa50-1e07-4ea2-be24-e951eeb56e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" Jul 16 21:57:15.167305 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:15.167283 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcbmt\" (UniqueName: \"kubernetes.io/projected/6826aa50-1e07-4ea2-be24-e951eeb56e21-kube-api-access-dcbmt\") pod \"maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht\" (UID: \"6826aa50-1e07-4ea2-be24-e951eeb56e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" Jul 16 21:57:15.261357 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:15.261295 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" Jul 16 21:57:15.404882 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:15.404858 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht"] Jul 16 21:57:15.405510 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:57:15.405479 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6826aa50_1e07_4ea2_be24_e951eeb56e21.slice/crio-b17fc2f2077d1c1d77753eb2ba90a441f4c046bf30a95c389bc836466fa8d905 WatchSource:0}: Error finding container b17fc2f2077d1c1d77753eb2ba90a441f4c046bf30a95c389bc836466fa8d905: Status 404 returned error can't find the container with id b17fc2f2077d1c1d77753eb2ba90a441f4c046bf30a95c389bc836466fa8d905 Jul 16 21:57:15.506595 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:15.506565 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" event={"ID":"6826aa50-1e07-4ea2-be24-e951eeb56e21","Type":"ContainerStarted","Data":"b17fc2f2077d1c1d77753eb2ba90a441f4c046bf30a95c389bc836466fa8d905"} Jul 16 21:57:16.513283 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:16.513246 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" event={"ID":"6826aa50-1e07-4ea2-be24-e951eeb56e21","Type":"ContainerStarted","Data":"8e3fc58b50e8e3c81ec3bde7c6ceb5e411e9b21980c91e1424ee8ff7224aa06d"} Jul 16 21:57:16.513741 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:16.513500 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" Jul 16 21:57:16.543842 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:16.543795 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" podStartSLOduration=2.086077612 podStartE2EDuration="2.543778757s" totalImagesPullingTime="457.701145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:16.542452776 +0000 UTC m=+2852.717560969" watchObservedRunningTime="2026-07-16 21:57:16.543778757 +0000 UTC m=+2852.718886948" Jul 16 21:57:21.445307 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:21.445276 2577 scope.go:117] "RemoveContainer" containerID="0e5cd0dca5626f9d40cc36e021bc3b6924cf944cc5d448194eff01fd6e17dc48" Jul 16 21:57:22.524118 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:22.524089 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" Jul 16 21:57:22.546985 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:22.546947 2577 generic.go:356] "Generic (PLEG): container finished" podID="d62808ce-0937-4973-91f4-80e161074555" containerID="fb62bc557c1f947a8a7207ac6cf96dad142a4c19c32924a3c3abdddd8115dfd0" exitCode=6 Jul 16 21:57:22.547163 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:22.547021 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd" event={"ID":"d62808ce-0937-4973-91f4-80e161074555","Type":"ContainerDied","Data":"fb62bc557c1f947a8a7207ac6cf96dad142a4c19c32924a3c3abdddd8115dfd0"} Jul 16 21:57:22.547163 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:22.547104 2577 scope.go:117] "RemoveContainer" containerID="0e5cd0dca5626f9d40cc36e021bc3b6924cf944cc5d448194eff01fd6e17dc48" Jul 16 21:57:22.547444 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:22.547423 2577 scope.go:117] "RemoveContainer" containerID="fb62bc557c1f947a8a7207ac6cf96dad142a4c19c32924a3c3abdddd8115dfd0" Jul 16 21:57:22.547830 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:57:22.547674 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd_odh-ai-gateway-infra(d62808ce-0937-4973-91f4-80e161074555)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd" podUID="d62808ce-0937-4973-91f4-80e161074555" Jul 16 21:57:23.603086 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:23.603048 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd"] Jul 16 21:57:23.755081 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:23.755055 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd" Jul 16 21:57:23.826856 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:23.826823 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62808ce-0937-4973-91f4-80e161074555-maas-api-service-ca\") pod \"d62808ce-0937-4973-91f4-80e161074555\" (UID: \"d62808ce-0937-4973-91f4-80e161074555\") " Jul 16 21:57:23.827215 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:23.827171 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62808ce-0937-4973-91f4-80e161074555-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d62808ce-0937-4973-91f4-80e161074555" (UID: "d62808ce-0937-4973-91f4-80e161074555"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:57:23.927645 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:23.927608 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62808ce-0937-4973-91f4-80e161074555-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:57:24.563880 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:24.563852 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd" Jul 16 21:57:24.564055 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:24.563901 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd" event={"ID":"d62808ce-0937-4973-91f4-80e161074555","Type":"ContainerDied","Data":"367a8c02542ad075f96e3422d914030071d3331759b6a31b4d9c48e190deb710"} Jul 16 21:57:24.564055 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:24.563963 2577 scope.go:117] "RemoveContainer" containerID="fb62bc557c1f947a8a7207ac6cf96dad142a4c19c32924a3c3abdddd8115dfd0" Jul 16 21:57:24.587030 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:24.586989 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd"] Jul 16 21:57:24.591749 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:24.591684 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-rxstd"] Jul 16 21:57:26.451577 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:26.451539 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d62808ce-0937-4973-91f4-80e161074555" path="/var/lib/kubelet/pods/d62808ce-0937-4973-91f4-80e161074555/volumes" Jul 16 21:57:27.582089 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:27.582061 2577 generic.go:356] "Generic (PLEG): container finished" podID="a20bca18-8725-49cd-8471-ff4a2f04aa8b" containerID="5f92ec3cf32418a4b59c0c62e87e8b01c39a8949d670de2ca047608f49103cad" exitCode=0 Jul 16 21:57:27.582504 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:27.582118 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" event={"ID":"a20bca18-8725-49cd-8471-ff4a2f04aa8b","Type":"ContainerDied","Data":"5f92ec3cf32418a4b59c0c62e87e8b01c39a8949d670de2ca047608f49103cad"} Jul 16 21:57:27.582578 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:27.582504 2577 scope.go:117] "RemoveContainer" containerID="5f92ec3cf32418a4b59c0c62e87e8b01c39a8949d670de2ca047608f49103cad" Jul 16 21:57:27.583469 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:27.583443 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:57:27.583524 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:27.583506 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:57:27.583564 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:27.583552 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:57:28.610456 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:28.610419 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" event={"ID":"a20bca18-8725-49cd-8471-ff4a2f04aa8b","Type":"ContainerStarted","Data":"9c393de72a170f88bd3d33652fd6e9410ee195e2d480f9c411e1196a4360bad1"} Jul 16 21:57:28.968813 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:28.968783 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:28.969686 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:28.969661 2577 patch_prober.go:28] interesting pod/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Jul 16 21:57:28.969810 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:28.969709 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" podUID="a20bca18-8725-49cd-8471-ff4a2f04aa8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Jul 16 21:57:29.500729 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:29.500667 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:29.968633 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:29.968597 2577 patch_prober.go:28] interesting pod/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Jul 16 21:57:29.969028 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:29.968651 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" podUID="a20bca18-8725-49cd-8471-ff4a2f04aa8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Jul 16 21:57:30.968743 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:30.968706 2577 patch_prober.go:28] interesting pod/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Jul 16 21:57:30.969178 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:30.968761 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" podUID="a20bca18-8725-49cd-8471-ff4a2f04aa8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Jul 16 21:57:31.246488 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.246364 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j"] Jul 16 21:57:31.246916 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.246898 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d62808ce-0937-4973-91f4-80e161074555" containerName="revoke-keys" Jul 16 21:57:31.246916 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.246917 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d62808ce-0937-4973-91f4-80e161074555" containerName="revoke-keys" Jul 16 21:57:31.247120 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.246955 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d62808ce-0937-4973-91f4-80e161074555" containerName="revoke-keys" Jul 16 21:57:31.247120 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.246972 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d62808ce-0937-4973-91f4-80e161074555" containerName="revoke-keys" Jul 16 21:57:31.247120 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.247060 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d62808ce-0937-4973-91f4-80e161074555" containerName="revoke-keys" Jul 16 21:57:31.247120 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.247077 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d62808ce-0937-4973-91f4-80e161074555" containerName="revoke-keys" Jul 16 21:57:31.250897 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.250877 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.258379 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.258359 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f53605\"/\"kube-root-ca.crt\"" Jul 16 21:57:31.259396 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.259381 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f53605\"/\"default-dockercfg-fspp9\"" Jul 16 21:57:31.259470 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.259411 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f53605\"/\"test-model-f53605-kserve-self-signed-certs\"" Jul 16 21:57:31.272197 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.272157 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j"] Jul 16 21:57:31.274108 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.274083 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f53605\"/\"openshift-service-ca.crt\"" Jul 16 21:57:31.292553 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.292518 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbl5m\" (UniqueName: \"kubernetes.io/projected/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-kube-api-access-kbl5m\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.292833 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.292797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-model-cache\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.293044 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.293011 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-dshm\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.293243 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.293224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-tls-certs\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.293515 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.293480 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-tmp-dir\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.293687 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.293664 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-home\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.396746 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.396667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-dshm\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.397066 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.397034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-tls-certs\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.397209 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.397092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-tmp-dir\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.397209 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.397122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-home\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.397328 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.397205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbl5m\" (UniqueName: \"kubernetes.io/projected/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-kube-api-access-kbl5m\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.398040 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.397596 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-tmp-dir\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.398040 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.397655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-model-cache\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.398040 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.397796 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-home\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.398040 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.398000 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-model-cache\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.400988 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.400947 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-tls-certs\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.402395 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.402374 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-dshm\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.445733 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.445704 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbl5m\" (UniqueName: \"kubernetes.io/projected/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-kube-api-access-kbl5m\") pod \"test-model-f53605-kserve-bf996567f-59h2j\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.560078 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.559984 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:31.761223 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.761171 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j"] Jul 16 21:57:31.761324 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:57:31.761218 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd15cae1f_9cf0_4ab3_9fe3_71b5f699c40a.slice/crio-55b7dc6f07df65e30ab80b33c68633936d4fe1fb62af1f5a095d856bcc2c4592 WatchSource:0}: Error finding container 55b7dc6f07df65e30ab80b33c68633936d4fe1fb62af1f5a095d856bcc2c4592: Status 404 returned error can't find the container with id 55b7dc6f07df65e30ab80b33c68633936d4fe1fb62af1f5a095d856bcc2c4592 Jul 16 21:57:31.968847 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.968807 2577 patch_prober.go:28] interesting pod/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Jul 16 21:57:31.969284 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:31.968862 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" podUID="a20bca18-8725-49cd-8471-ff4a2f04aa8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Jul 16 21:57:32.637132 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:32.637095 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" event={"ID":"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a","Type":"ContainerStarted","Data":"6d1ef0feae6b7f3a8dae63dcb8d5c01eaeecc3f5116bacb5cc7fbfe63e2b5dba"} Jul 16 21:57:32.637333 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:32.637147 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" event={"ID":"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a","Type":"ContainerStarted","Data":"55b7dc6f07df65e30ab80b33c68633936d4fe1fb62af1f5a095d856bcc2c4592"} Jul 16 21:57:32.972625 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:32.972549 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:32.980220 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:32.980167 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:57:33.022570 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:33.022517 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" podStartSLOduration=2.022502541 podStartE2EDuration="2.022502541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:32.671488115 +0000 UTC m=+2868.846596307" watchObservedRunningTime="2026-07-16 21:57:33.022502541 +0000 UTC m=+2869.197610764" Jul 16 21:57:41.560353 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:41.560263 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:41.560353 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:41.560322 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:41.572517 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:41.572486 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:57:41.696605 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:57:41.696576 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:58:23.151210 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:23.151146 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j"] Jul 16 21:58:34.880437 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:34.880397 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5"] Jul 16 21:58:34.880978 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:34.880616 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" podUID="cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc" containerName="maas-api" containerID="cri-o://2cf1bbad3abee82b00d39dd3dae83886f2b51ee1bcc67ddc4ad085b75fee9d00" gracePeriod=30 Jul 16 21:58:35.128737 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.128714 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" Jul 16 21:58:35.262144 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.262056 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-maas-api-tls\") pod \"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc\" (UID: \"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc\") " Jul 16 21:58:35.262144 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.262104 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-openshift-service-ca\") pod \"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc\" (UID: \"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc\") " Jul 16 21:58:35.262406 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.262146 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4gcn\" (UniqueName: \"kubernetes.io/projected/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-kube-api-access-l4gcn\") pod \"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc\" (UID: \"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc\") " Jul 16 21:58:35.262503 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.262479 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc" (UID: "cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:58:35.264099 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.264069 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc" (UID: "cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:58:35.264240 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.264143 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-kube-api-access-l4gcn" (OuterVolumeSpecName: "kube-api-access-l4gcn") pod "cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc" (UID: "cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc"). InnerVolumeSpecName "kube-api-access-l4gcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:58:35.363692 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.363655 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-maas-api-tls\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:58:35.363692 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.363689 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-openshift-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:58:35.363692 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.363700 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4gcn\" (UniqueName: \"kubernetes.io/projected/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc-kube-api-access-l4gcn\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:58:35.963042 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.963006 2577 generic.go:356] "Generic (PLEG): container finished" podID="cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc" containerID="2cf1bbad3abee82b00d39dd3dae83886f2b51ee1bcc67ddc4ad085b75fee9d00" exitCode=0 Jul 16 21:58:35.963512 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.963052 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" Jul 16 21:58:35.963512 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.963081 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" event={"ID":"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc","Type":"ContainerDied","Data":"2cf1bbad3abee82b00d39dd3dae83886f2b51ee1bcc67ddc4ad085b75fee9d00"} Jul 16 21:58:35.963512 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.963134 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5" event={"ID":"cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc","Type":"ContainerDied","Data":"4bf657b8405958d2fab5c360b33e58cdf0689b1796dae1e189a84821a48c9b4d"} Jul 16 21:58:35.963512 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.963159 2577 scope.go:117] "RemoveContainer" containerID="2cf1bbad3abee82b00d39dd3dae83886f2b51ee1bcc67ddc4ad085b75fee9d00" Jul 16 21:58:35.973070 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.973050 2577 scope.go:117] "RemoveContainer" containerID="2cf1bbad3abee82b00d39dd3dae83886f2b51ee1bcc67ddc4ad085b75fee9d00" Jul 16 21:58:35.973352 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:58:35.973334 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf1bbad3abee82b00d39dd3dae83886f2b51ee1bcc67ddc4ad085b75fee9d00\": container with ID starting with 2cf1bbad3abee82b00d39dd3dae83886f2b51ee1bcc67ddc4ad085b75fee9d00 not found: ID does not exist" containerID="2cf1bbad3abee82b00d39dd3dae83886f2b51ee1bcc67ddc4ad085b75fee9d00" Jul 16 21:58:35.973415 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.973360 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf1bbad3abee82b00d39dd3dae83886f2b51ee1bcc67ddc4ad085b75fee9d00"} err="failed to get container status \"2cf1bbad3abee82b00d39dd3dae83886f2b51ee1bcc67ddc4ad085b75fee9d00\": rpc error: code = NotFound desc = could not find container \"2cf1bbad3abee82b00d39dd3dae83886f2b51ee1bcc67ddc4ad085b75fee9d00\": container with ID starting with 2cf1bbad3abee82b00d39dd3dae83886f2b51ee1bcc67ddc4ad085b75fee9d00 not found: ID does not exist" Jul 16 21:58:35.986986 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.986951 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5"] Jul 16 21:58:35.991449 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:35.991427 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f53605-68595969cc-mdfk5"] Jul 16 21:58:36.450942 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:36.450910 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc" path="/var/lib/kubelet/pods/cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc/volumes" Jul 16 21:58:38.171139 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.171095 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" podUID="d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a" containerName="main" containerID="cri-o://6d1ef0feae6b7f3a8dae63dcb8d5c01eaeecc3f5116bacb5cc7fbfe63e2b5dba" gracePeriod=45 Jul 16 21:58:38.418223 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.418197 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:58:38.492670 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.492589 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-tmp-dir\") pod \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " Jul 16 21:58:38.492670 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.492637 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbl5m\" (UniqueName: \"kubernetes.io/projected/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-kube-api-access-kbl5m\") pod \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " Jul 16 21:58:38.492867 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.492680 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-tls-certs\") pod \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " Jul 16 21:58:38.492867 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.492730 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-dshm\") pod \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " Jul 16 21:58:38.492867 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.492767 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-model-cache\") pod \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " Jul 16 21:58:38.492867 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.492797 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-home\") pod \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\" (UID: \"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a\") " Jul 16 21:58:38.493025 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.492896 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a" (UID: "d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:58:38.493095 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.493050 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a" (UID: "d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:58:38.493163 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.493096 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-tmp-dir\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:58:38.493163 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.493098 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-home" (OuterVolumeSpecName: "home") pod "d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a" (UID: "d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:58:38.494965 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.494935 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-dshm" (OuterVolumeSpecName: "dshm") pod "d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a" (UID: "d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:58:38.495108 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.494988 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a" (UID: "d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:58:38.495108 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.494994 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-kube-api-access-kbl5m" (OuterVolumeSpecName: "kube-api-access-kbl5m") pod "d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a" (UID: "d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a"). InnerVolumeSpecName "kube-api-access-kbl5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:58:38.593694 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.593660 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-tls-certs\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:58:38.593694 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.593690 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-dshm\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:58:38.593694 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.593698 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-model-cache\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:58:38.593694 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.593708 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-home\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:58:38.593974 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.593715 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbl5m\" (UniqueName: \"kubernetes.io/projected/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a-kube-api-access-kbl5m\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:58:38.980829 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.980791 2577 generic.go:356] "Generic (PLEG): container finished" podID="d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a" containerID="6d1ef0feae6b7f3a8dae63dcb8d5c01eaeecc3f5116bacb5cc7fbfe63e2b5dba" exitCode=0 Jul 16 21:58:38.981007 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.980851 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" Jul 16 21:58:38.981007 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.980877 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" event={"ID":"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a","Type":"ContainerDied","Data":"6d1ef0feae6b7f3a8dae63dcb8d5c01eaeecc3f5116bacb5cc7fbfe63e2b5dba"} Jul 16 21:58:38.981007 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.980924 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j" event={"ID":"d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a","Type":"ContainerDied","Data":"55b7dc6f07df65e30ab80b33c68633936d4fe1fb62af1f5a095d856bcc2c4592"} Jul 16 21:58:38.981007 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.980942 2577 scope.go:117] "RemoveContainer" containerID="6d1ef0feae6b7f3a8dae63dcb8d5c01eaeecc3f5116bacb5cc7fbfe63e2b5dba" Jul 16 21:58:38.990258 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.990239 2577 scope.go:117] "RemoveContainer" containerID="6d1ef0feae6b7f3a8dae63dcb8d5c01eaeecc3f5116bacb5cc7fbfe63e2b5dba" Jul 16 21:58:38.990565 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:58:38.990544 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1ef0feae6b7f3a8dae63dcb8d5c01eaeecc3f5116bacb5cc7fbfe63e2b5dba\": container with ID starting with 6d1ef0feae6b7f3a8dae63dcb8d5c01eaeecc3f5116bacb5cc7fbfe63e2b5dba not found: ID does not exist" containerID="6d1ef0feae6b7f3a8dae63dcb8d5c01eaeecc3f5116bacb5cc7fbfe63e2b5dba" Jul 16 21:58:38.990678 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:38.990572 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1ef0feae6b7f3a8dae63dcb8d5c01eaeecc3f5116bacb5cc7fbfe63e2b5dba"} err="failed to get container status \"6d1ef0feae6b7f3a8dae63dcb8d5c01eaeecc3f5116bacb5cc7fbfe63e2b5dba\": rpc error: code = NotFound desc = could not find container \"6d1ef0feae6b7f3a8dae63dcb8d5c01eaeecc3f5116bacb5cc7fbfe63e2b5dba\": container with ID starting with 6d1ef0feae6b7f3a8dae63dcb8d5c01eaeecc3f5116bacb5cc7fbfe63e2b5dba not found: ID does not exist" Jul 16 21:58:39.005831 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:39.005801 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j"] Jul 16 21:58:39.010943 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:39.010913 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f53605/test-model-f53605-kserve-bf996567f-59h2j"] Jul 16 21:58:40.450606 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:40.450573 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a" path="/var/lib/kubelet/pods/d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a/volumes" Jul 16 21:58:50.694863 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:50.694833 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs"] Jul 16 21:58:50.695276 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:50.695206 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a" containerName="main" Jul 16 21:58:50.695276 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:50.695219 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a" containerName="main" Jul 16 21:58:50.695276 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:50.695232 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc" containerName="maas-api" Jul 16 21:58:50.695276 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:50.695238 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc" containerName="maas-api" Jul 16 21:58:50.695276 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:50.695253 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d62808ce-0937-4973-91f4-80e161074555" containerName="revoke-keys" Jul 16 21:58:50.695276 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:50.695259 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d62808ce-0937-4973-91f4-80e161074555" containerName="revoke-keys" Jul 16 21:58:50.695476 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:50.695316 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd8ec15a-d6c8-44f6-822c-c1893eb1ebcc" containerName="maas-api" Jul 16 21:58:50.695476 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:50.695323 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d15cae1f-9cf0-4ab3-9fe3-71b5f699c40a" containerName="main" Jul 16 21:58:50.695476 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:50.695330 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d62808ce-0937-4973-91f4-80e161074555" containerName="revoke-keys" Jul 16 21:58:50.699386 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:50.699371 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs" Jul 16 21:58:50.702759 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:50.702739 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:58:50.709002 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:50.708984 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs"] Jul 16 21:58:50.802565 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:50.802530 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b279c32c-76c4-42d4-98c5-ce5f662acd45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs\" (UID: \"b279c32c-76c4-42d4-98c5-ce5f662acd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs" Jul 16 21:58:50.903880 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:50.903849 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b279c32c-76c4-42d4-98c5-ce5f662acd45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs\" (UID: \"b279c32c-76c4-42d4-98c5-ce5f662acd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs" Jul 16 21:58:50.904489 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:50.904471 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b279c32c-76c4-42d4-98c5-ce5f662acd45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs\" (UID: \"b279c32c-76c4-42d4-98c5-ce5f662acd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs" Jul 16 21:58:51.010671 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:51.010594 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs" Jul 16 21:58:51.362067 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:51.362045 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs"] Jul 16 21:58:51.363723 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:58:51.363693 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb279c32c_76c4_42d4_98c5_ce5f662acd45.slice/crio-32a1a9f9f036cfe9742f807288cc586374ff19e68fa29b2f9b1ee72b50a678a4 WatchSource:0}: Error finding container 32a1a9f9f036cfe9742f807288cc586374ff19e68fa29b2f9b1ee72b50a678a4: Status 404 returned error can't find the container with id 32a1a9f9f036cfe9742f807288cc586374ff19e68fa29b2f9b1ee72b50a678a4 Jul 16 21:58:52.053786 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:52.053758 2577 generic.go:356] "Generic (PLEG): container finished" podID="b279c32c-76c4-42d4-98c5-ce5f662acd45" containerID="857993a8909f6ab2b88768e935e8e584e318344f6a8617d0784d972310b46236" exitCode=0 Jul 16 21:58:52.054148 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:52.053810 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs" event={"ID":"b279c32c-76c4-42d4-98c5-ce5f662acd45","Type":"ContainerDied","Data":"857993a8909f6ab2b88768e935e8e584e318344f6a8617d0784d972310b46236"} Jul 16 21:58:52.054148 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:52.053871 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs" event={"ID":"b279c32c-76c4-42d4-98c5-ce5f662acd45","Type":"ContainerStarted","Data":"32a1a9f9f036cfe9742f807288cc586374ff19e68fa29b2f9b1ee72b50a678a4"} Jul 16 21:58:53.195531 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:53.195507 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs" Jul 16 21:58:53.323223 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:53.323101 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b279c32c-76c4-42d4-98c5-ce5f662acd45-maas-api-service-ca\") pod \"b279c32c-76c4-42d4-98c5-ce5f662acd45\" (UID: \"b279c32c-76c4-42d4-98c5-ce5f662acd45\") " Jul 16 21:58:53.323504 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:53.323481 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b279c32c-76c4-42d4-98c5-ce5f662acd45-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b279c32c-76c4-42d4-98c5-ce5f662acd45" (UID: "b279c32c-76c4-42d4-98c5-ce5f662acd45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:58:53.424677 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:53.424640 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b279c32c-76c4-42d4-98c5-ce5f662acd45-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:58:54.065992 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:54.065965 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs" Jul 16 21:58:54.066208 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:54.066015 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs" event={"ID":"b279c32c-76c4-42d4-98c5-ce5f662acd45","Type":"ContainerDied","Data":"32a1a9f9f036cfe9742f807288cc586374ff19e68fa29b2f9b1ee72b50a678a4"} Jul 16 21:58:54.066208 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:58:54.066070 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32a1a9f9f036cfe9742f807288cc586374ff19e68fa29b2f9b1ee72b50a678a4" Jul 16 21:59:01.141917 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:01.141881 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht"] Jul 16 21:59:01.142381 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:01.142096 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" podUID="6826aa50-1e07-4ea2-be24-e951eeb56e21" containerName="maas-api" containerID="cri-o://8e3fc58b50e8e3c81ec3bde7c6ceb5e411e9b21980c91e1424ee8ff7224aa06d" gracePeriod=30 Jul 16 21:59:01.384430 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:01.384407 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" Jul 16 21:59:01.397911 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:01.397864 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcbmt\" (UniqueName: \"kubernetes.io/projected/6826aa50-1e07-4ea2-be24-e951eeb56e21-kube-api-access-dcbmt\") pod \"6826aa50-1e07-4ea2-be24-e951eeb56e21\" (UID: \"6826aa50-1e07-4ea2-be24-e951eeb56e21\") " Jul 16 21:59:01.398008 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:01.397919 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6826aa50-1e07-4ea2-be24-e951eeb56e21-maas-api-tls\") pod \"6826aa50-1e07-4ea2-be24-e951eeb56e21\" (UID: \"6826aa50-1e07-4ea2-be24-e951eeb56e21\") " Jul 16 21:59:01.398058 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:01.398029 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6826aa50-1e07-4ea2-be24-e951eeb56e21-openshift-service-ca\") pod \"6826aa50-1e07-4ea2-be24-e951eeb56e21\" (UID: \"6826aa50-1e07-4ea2-be24-e951eeb56e21\") " Jul 16 21:59:01.398394 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:01.398372 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6826aa50-1e07-4ea2-be24-e951eeb56e21-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6826aa50-1e07-4ea2-be24-e951eeb56e21" (UID: "6826aa50-1e07-4ea2-be24-e951eeb56e21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:59:01.399815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:01.399789 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6826aa50-1e07-4ea2-be24-e951eeb56e21-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6826aa50-1e07-4ea2-be24-e951eeb56e21" (UID: "6826aa50-1e07-4ea2-be24-e951eeb56e21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:59:01.399902 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:01.399845 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6826aa50-1e07-4ea2-be24-e951eeb56e21-kube-api-access-dcbmt" (OuterVolumeSpecName: "kube-api-access-dcbmt") pod "6826aa50-1e07-4ea2-be24-e951eeb56e21" (UID: "6826aa50-1e07-4ea2-be24-e951eeb56e21"). InnerVolumeSpecName "kube-api-access-dcbmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:59:01.499024 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:01.498989 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6826aa50-1e07-4ea2-be24-e951eeb56e21-openshift-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:59:01.499024 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:01.499018 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcbmt\" (UniqueName: \"kubernetes.io/projected/6826aa50-1e07-4ea2-be24-e951eeb56e21-kube-api-access-dcbmt\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:59:01.499024 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:01.499029 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6826aa50-1e07-4ea2-be24-e951eeb56e21-maas-api-tls\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:59:02.108631 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:02.108588 2577 generic.go:356] "Generic (PLEG): container finished" podID="6826aa50-1e07-4ea2-be24-e951eeb56e21" containerID="8e3fc58b50e8e3c81ec3bde7c6ceb5e411e9b21980c91e1424ee8ff7224aa06d" exitCode=0 Jul 16 21:59:02.108631 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:02.108630 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" event={"ID":"6826aa50-1e07-4ea2-be24-e951eeb56e21","Type":"ContainerDied","Data":"8e3fc58b50e8e3c81ec3bde7c6ceb5e411e9b21980c91e1424ee8ff7224aa06d"} Jul 16 21:59:02.108843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:02.108680 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" event={"ID":"6826aa50-1e07-4ea2-be24-e951eeb56e21","Type":"ContainerDied","Data":"b17fc2f2077d1c1d77753eb2ba90a441f4c046bf30a95c389bc836466fa8d905"} Jul 16 21:59:02.108843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:02.108701 2577 scope.go:117] "RemoveContainer" containerID="8e3fc58b50e8e3c81ec3bde7c6ceb5e411e9b21980c91e1424ee8ff7224aa06d" Jul 16 21:59:02.108843 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:02.108634 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht" Jul 16 21:59:02.118295 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:02.118274 2577 scope.go:117] "RemoveContainer" containerID="8e3fc58b50e8e3c81ec3bde7c6ceb5e411e9b21980c91e1424ee8ff7224aa06d" Jul 16 21:59:02.118545 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:59:02.118525 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3fc58b50e8e3c81ec3bde7c6ceb5e411e9b21980c91e1424ee8ff7224aa06d\": container with ID starting with 8e3fc58b50e8e3c81ec3bde7c6ceb5e411e9b21980c91e1424ee8ff7224aa06d not found: ID does not exist" containerID="8e3fc58b50e8e3c81ec3bde7c6ceb5e411e9b21980c91e1424ee8ff7224aa06d" Jul 16 21:59:02.118616 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:02.118551 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3fc58b50e8e3c81ec3bde7c6ceb5e411e9b21980c91e1424ee8ff7224aa06d"} err="failed to get container status \"8e3fc58b50e8e3c81ec3bde7c6ceb5e411e9b21980c91e1424ee8ff7224aa06d\": rpc error: code = NotFound desc = could not find container \"8e3fc58b50e8e3c81ec3bde7c6ceb5e411e9b21980c91e1424ee8ff7224aa06d\": container with ID starting with 8e3fc58b50e8e3c81ec3bde7c6ceb5e411e9b21980c91e1424ee8ff7224aa06d not found: ID does not exist" Jul 16 21:59:02.132015 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:02.131992 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht"] Jul 16 21:59:02.135726 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:02.135709 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6e4744-798dc8bd5c-djvht"] Jul 16 21:59:02.451022 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:02.450990 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6826aa50-1e07-4ea2-be24-e951eeb56e21" path="/var/lib/kubelet/pods/6826aa50-1e07-4ea2-be24-e951eeb56e21/volumes" Jul 16 21:59:07.362010 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:07.361932 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr"] Jul 16 21:59:07.362461 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:07.362375 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b279c32c-76c4-42d4-98c5-ce5f662acd45" containerName="revoke-keys" Jul 16 21:59:07.362461 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:07.362390 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b279c32c-76c4-42d4-98c5-ce5f662acd45" containerName="revoke-keys" Jul 16 21:59:07.362461 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:07.362403 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6826aa50-1e07-4ea2-be24-e951eeb56e21" containerName="maas-api" Jul 16 21:59:07.362461 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:07.362408 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6826aa50-1e07-4ea2-be24-e951eeb56e21" containerName="maas-api" Jul 16 21:59:07.362461 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:07.362463 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b279c32c-76c4-42d4-98c5-ce5f662acd45" containerName="revoke-keys" Jul 16 21:59:07.362629 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:07.362474 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6826aa50-1e07-4ea2-be24-e951eeb56e21" containerName="maas-api" Jul 16 21:59:07.366982 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:07.366959 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" Jul 16 21:59:07.369895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:07.369872 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:59:07.370029 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:07.369959 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 21:59:07.374533 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:07.374177 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr"] Jul 16 21:59:07.447202 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:07.447159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e88f7139-ef46-468d-b503-efd26517102f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr\" (UID: \"e88f7139-ef46-468d-b503-efd26517102f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" Jul 16 21:59:07.548064 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:07.548035 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e88f7139-ef46-468d-b503-efd26517102f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr\" (UID: \"e88f7139-ef46-468d-b503-efd26517102f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" Jul 16 21:59:07.548618 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:07.548601 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e88f7139-ef46-468d-b503-efd26517102f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr\" (UID: \"e88f7139-ef46-468d-b503-efd26517102f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" Jul 16 21:59:07.678689 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:07.678671 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" Jul 16 21:59:07.806101 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:07.806076 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr"] Jul 16 21:59:07.807730 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:59:07.807705 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88f7139_ef46_468d_b503_efd26517102f.slice/crio-441cdeffade798ec7d18f12715209f9701bbfda05024846614b6bcf5f4b8b477 WatchSource:0}: Error finding container 441cdeffade798ec7d18f12715209f9701bbfda05024846614b6bcf5f4b8b477: Status 404 returned error can't find the container with id 441cdeffade798ec7d18f12715209f9701bbfda05024846614b6bcf5f4b8b477 Jul 16 21:59:08.141373 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:08.141343 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" event={"ID":"e88f7139-ef46-468d-b503-efd26517102f","Type":"ContainerStarted","Data":"2270d041405bdaf0e9fc4ceac4d89d1698d82710d5d63ed83fd063a6f81a4aea"} Jul 16 21:59:08.141478 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:08.141401 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" event={"ID":"e88f7139-ef46-468d-b503-efd26517102f","Type":"ContainerStarted","Data":"441cdeffade798ec7d18f12715209f9701bbfda05024846614b6bcf5f4b8b477"} Jul 16 21:59:08.160265 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:08.160220 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" podStartSLOduration=1.1602063 podStartE2EDuration="1.1602063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:59:08.157733253 +0000 UTC m=+2964.332841444" watchObservedRunningTime="2026-07-16 21:59:08.1602063 +0000 UTC m=+2964.335314482" Jul 16 21:59:09.148005 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:09.147974 2577 generic.go:356] "Generic (PLEG): container finished" podID="e88f7139-ef46-468d-b503-efd26517102f" containerID="2270d041405bdaf0e9fc4ceac4d89d1698d82710d5d63ed83fd063a6f81a4aea" exitCode=6 Jul 16 21:59:09.148428 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:09.148067 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" event={"ID":"e88f7139-ef46-468d-b503-efd26517102f","Type":"ContainerDied","Data":"2270d041405bdaf0e9fc4ceac4d89d1698d82710d5d63ed83fd063a6f81a4aea"} Jul 16 21:59:09.148428 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:09.148247 2577 scope.go:117] "RemoveContainer" containerID="2270d041405bdaf0e9fc4ceac4d89d1698d82710d5d63ed83fd063a6f81a4aea" Jul 16 21:59:10.155118 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:10.155080 2577 generic.go:356] "Generic (PLEG): container finished" podID="e88f7139-ef46-468d-b503-efd26517102f" containerID="b0468345851d0ff208951101616c12e6d367d3f1e4df0c6e856a1d5d50d1ce4f" exitCode=6 Jul 16 21:59:10.155594 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:10.155153 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" event={"ID":"e88f7139-ef46-468d-b503-efd26517102f","Type":"ContainerDied","Data":"b0468345851d0ff208951101616c12e6d367d3f1e4df0c6e856a1d5d50d1ce4f"} Jul 16 21:59:10.155594 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:10.155265 2577 scope.go:117] "RemoveContainer" containerID="2270d041405bdaf0e9fc4ceac4d89d1698d82710d5d63ed83fd063a6f81a4aea" Jul 16 21:59:10.155594 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:10.155467 2577 scope.go:117] "RemoveContainer" containerID="b0468345851d0ff208951101616c12e6d367d3f1e4df0c6e856a1d5d50d1ce4f" Jul 16 21:59:10.155735 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:59:10.155707 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr_odh-ai-gateway-infra(e88f7139-ef46-468d-b503-efd26517102f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" podUID="e88f7139-ef46-468d-b503-efd26517102f" Jul 16 21:59:11.161517 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:11.161493 2577 scope.go:117] "RemoveContainer" containerID="b0468345851d0ff208951101616c12e6d367d3f1e4df0c6e856a1d5d50d1ce4f" Jul 16 21:59:11.161880 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:59:11.161644 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr_odh-ai-gateway-infra(e88f7139-ef46-468d-b503-efd26517102f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" podUID="e88f7139-ef46-468d-b503-efd26517102f" Jul 16 21:59:16.484602 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:16.484562 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf"] Jul 16 21:59:16.485073 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:16.484811 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" podUID="a20bca18-8725-49cd-8471-ff4a2f04aa8b" containerName="istio-proxy" containerID="cri-o://9c393de72a170f88bd3d33652fd6e9410ee195e2d480f9c411e1196a4360bad1" gracePeriod=30 Jul 16 21:59:17.975404 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:17.975361 2577 patch_prober.go:28] interesting pod/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 21:59:17.975780 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:17.975441 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" podUID="a20bca18-8725-49cd-8471-ff4a2f04aa8b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 21:59:21.132104 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.132070 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq"] Jul 16 21:59:21.136722 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.136706 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" Jul 16 21:59:21.141913 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.141872 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f90d76-serving-cert\"" Jul 16 21:59:21.142077 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.141930 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xcg5c\"" Jul 16 21:59:21.142572 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.142555 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 21:59:21.153538 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.153520 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq"] Jul 16 21:59:21.266179 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.266151 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hvmh\" (UniqueName: \"kubernetes.io/projected/a6c81b8a-5983-4649-aa15-b93b2bb828a9-kube-api-access-4hvmh\") pod \"maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq\" (UID: \"a6c81b8a-5983-4649-aa15-b93b2bb828a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" Jul 16 21:59:21.266375 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.266206 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c81b8a-5983-4649-aa15-b93b2bb828a9-openshift-service-ca\") pod \"maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq\" (UID: \"a6c81b8a-5983-4649-aa15-b93b2bb828a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" Jul 16 21:59:21.266375 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.266308 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c81b8a-5983-4649-aa15-b93b2bb828a9-maas-api-tls\") pod \"maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq\" (UID: \"a6c81b8a-5983-4649-aa15-b93b2bb828a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" Jul 16 21:59:21.367479 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.367444 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hvmh\" (UniqueName: \"kubernetes.io/projected/a6c81b8a-5983-4649-aa15-b93b2bb828a9-kube-api-access-4hvmh\") pod \"maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq\" (UID: \"a6c81b8a-5983-4649-aa15-b93b2bb828a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" Jul 16 21:59:21.367655 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.367490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c81b8a-5983-4649-aa15-b93b2bb828a9-openshift-service-ca\") pod \"maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq\" (UID: \"a6c81b8a-5983-4649-aa15-b93b2bb828a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" Jul 16 21:59:21.367655 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.367549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c81b8a-5983-4649-aa15-b93b2bb828a9-maas-api-tls\") pod \"maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq\" (UID: \"a6c81b8a-5983-4649-aa15-b93b2bb828a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" Jul 16 21:59:21.367787 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:59:21.367717 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-serving-cert: secret "maas-api-e2e-api-a-f90d76-serving-cert" not found Jul 16 21:59:21.367842 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:59:21.367817 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6c81b8a-5983-4649-aa15-b93b2bb828a9-maas-api-tls podName:a6c81b8a-5983-4649-aa15-b93b2bb828a9 nodeName:}" failed. No retries permitted until 2026-07-16 21:59:21.867795242 +0000 UTC m=+2978.042903416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6c81b8a-5983-4649-aa15-b93b2bb828a9-maas-api-tls") pod "maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" (UID: "a6c81b8a-5983-4649-aa15-b93b2bb828a9") : secret "maas-api-e2e-api-a-f90d76-serving-cert" not found Jul 16 21:59:21.368133 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.368112 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c81b8a-5983-4649-aa15-b93b2bb828a9-openshift-service-ca\") pod \"maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq\" (UID: \"a6c81b8a-5983-4649-aa15-b93b2bb828a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" Jul 16 21:59:21.377834 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.377809 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hvmh\" (UniqueName: \"kubernetes.io/projected/a6c81b8a-5983-4649-aa15-b93b2bb828a9-kube-api-access-4hvmh\") pod \"maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq\" (UID: \"a6c81b8a-5983-4649-aa15-b93b2bb828a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" Jul 16 21:59:21.745101 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.745040 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:59:21.872547 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.872516 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istiod-ca-cert\") pod \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " Jul 16 21:59:21.872547 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.872551 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-workload-socket\") pod \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " Jul 16 21:59:21.872781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.872574 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-envoy\") pod \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " Jul 16 21:59:21.872781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.872627 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-credential-socket\") pod \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " Jul 16 21:59:21.872781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.872654 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-workload-certs\") pod \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " Jul 16 21:59:21.872781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.872682 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-podinfo\") pod \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " Jul 16 21:59:21.872781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.872738 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-token\") pod \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " Jul 16 21:59:21.872781 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.872763 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-data\") pod \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " Jul 16 21:59:21.873097 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.872781 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a20bca18-8725-49cd-8471-ff4a2f04aa8b" (UID: "a20bca18-8725-49cd-8471-ff4a2f04aa8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:59:21.873097 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.872792 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvpdw\" (UniqueName: \"kubernetes.io/projected/a20bca18-8725-49cd-8471-ff4a2f04aa8b-kube-api-access-hvpdw\") pod \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\" (UID: \"a20bca18-8725-49cd-8471-ff4a2f04aa8b\") " Jul 16 21:59:21.873097 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.872985 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a20bca18-8725-49cd-8471-ff4a2f04aa8b" (UID: "a20bca18-8725-49cd-8471-ff4a2f04aa8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:59:21.873097 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.873027 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a20bca18-8725-49cd-8471-ff4a2f04aa8b" (UID: "a20bca18-8725-49cd-8471-ff4a2f04aa8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:59:21.873097 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.873065 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c81b8a-5983-4649-aa15-b93b2bb828a9-maas-api-tls\") pod \"maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq\" (UID: \"a6c81b8a-5983-4649-aa15-b93b2bb828a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" Jul 16 21:59:21.873412 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.873240 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istiod-ca-cert\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:59:21.873412 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.873257 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-workload-socket\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:59:21.873412 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.873270 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-credential-socket\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:59:21.873412 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.873323 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a20bca18-8725-49cd-8471-ff4a2f04aa8b" (UID: "a20bca18-8725-49cd-8471-ff4a2f04aa8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:59:21.873815 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.873782 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-data" (OuterVolumeSpecName: "istio-data") pod "a20bca18-8725-49cd-8471-ff4a2f04aa8b" (UID: "a20bca18-8725-49cd-8471-ff4a2f04aa8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:59:21.875144 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.875101 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a20bca18-8725-49cd-8471-ff4a2f04aa8b" (UID: "a20bca18-8725-49cd-8471-ff4a2f04aa8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 21:59:21.875372 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.875345 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-token" (OuterVolumeSpecName: "istio-token") pod "a20bca18-8725-49cd-8471-ff4a2f04aa8b" (UID: "a20bca18-8725-49cd-8471-ff4a2f04aa8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:59:21.875472 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.875401 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a20bca18-8725-49cd-8471-ff4a2f04aa8b" (UID: "a20bca18-8725-49cd-8471-ff4a2f04aa8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:59:21.875527 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.875463 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20bca18-8725-49cd-8471-ff4a2f04aa8b-kube-api-access-hvpdw" (OuterVolumeSpecName: "kube-api-access-hvpdw") pod "a20bca18-8725-49cd-8471-ff4a2f04aa8b" (UID: "a20bca18-8725-49cd-8471-ff4a2f04aa8b"). InnerVolumeSpecName "kube-api-access-hvpdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:59:21.876036 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.876017 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c81b8a-5983-4649-aa15-b93b2bb828a9-maas-api-tls\") pod \"maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq\" (UID: \"a6c81b8a-5983-4649-aa15-b93b2bb828a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" Jul 16 21:59:21.974705 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.974672 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-token\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:59:21.974861 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.974709 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-data\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:59:21.974861 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.974729 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvpdw\" (UniqueName: \"kubernetes.io/projected/a20bca18-8725-49cd-8471-ff4a2f04aa8b-kube-api-access-hvpdw\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:59:21.974861 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.974742 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-envoy\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:59:21.974861 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.974757 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a20bca18-8725-49cd-8471-ff4a2f04aa8b-workload-certs\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:59:21.974861 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:21.974769 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a20bca18-8725-49cd-8471-ff4a2f04aa8b-istio-podinfo\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:59:22.046877 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.046804 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" Jul 16 21:59:22.178396 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.178369 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq"] Jul 16 21:59:22.179336 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:59:22.179315 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c81b8a_5983_4649_aa15_b93b2bb828a9.slice/crio-6d40b3cf4690159e921a0f94e2576d5af59bfd7a5a753f2ebfd03fc2e641bb54 WatchSource:0}: Error finding container 6d40b3cf4690159e921a0f94e2576d5af59bfd7a5a753f2ebfd03fc2e641bb54: Status 404 returned error can't find the container with id 6d40b3cf4690159e921a0f94e2576d5af59bfd7a5a753f2ebfd03fc2e641bb54 Jul 16 21:59:22.180589 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.180573 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:59:22.220970 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.220937 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" event={"ID":"a6c81b8a-5983-4649-aa15-b93b2bb828a9","Type":"ContainerStarted","Data":"6d40b3cf4690159e921a0f94e2576d5af59bfd7a5a753f2ebfd03fc2e641bb54"} Jul 16 21:59:22.228892 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.228869 2577 generic.go:356] "Generic (PLEG): container finished" podID="a20bca18-8725-49cd-8471-ff4a2f04aa8b" containerID="9c393de72a170f88bd3d33652fd6e9410ee195e2d480f9c411e1196a4360bad1" exitCode=0 Jul 16 21:59:22.228972 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.228929 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" Jul 16 21:59:22.228972 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.228938 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" event={"ID":"a20bca18-8725-49cd-8471-ff4a2f04aa8b","Type":"ContainerDied","Data":"9c393de72a170f88bd3d33652fd6e9410ee195e2d480f9c411e1196a4360bad1"} Jul 16 21:59:22.229053 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.228997 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf" event={"ID":"a20bca18-8725-49cd-8471-ff4a2f04aa8b","Type":"ContainerDied","Data":"f39ea6a4956c811da3968c08ad1958b5c29397cfd050a6c9bf9dae56e158db33"} Jul 16 21:59:22.229053 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.229018 2577 scope.go:117] "RemoveContainer" containerID="9c393de72a170f88bd3d33652fd6e9410ee195e2d480f9c411e1196a4360bad1" Jul 16 21:59:22.238664 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.238646 2577 scope.go:117] "RemoveContainer" containerID="5f92ec3cf32418a4b59c0c62e87e8b01c39a8949d670de2ca047608f49103cad" Jul 16 21:59:22.246561 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.246545 2577 scope.go:117] "RemoveContainer" containerID="9c393de72a170f88bd3d33652fd6e9410ee195e2d480f9c411e1196a4360bad1" Jul 16 21:59:22.246801 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:59:22.246784 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c393de72a170f88bd3d33652fd6e9410ee195e2d480f9c411e1196a4360bad1\": container with ID starting with 9c393de72a170f88bd3d33652fd6e9410ee195e2d480f9c411e1196a4360bad1 not found: ID does not exist" containerID="9c393de72a170f88bd3d33652fd6e9410ee195e2d480f9c411e1196a4360bad1" Jul 16 21:59:22.246850 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.246809 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c393de72a170f88bd3d33652fd6e9410ee195e2d480f9c411e1196a4360bad1"} err="failed to get container status \"9c393de72a170f88bd3d33652fd6e9410ee195e2d480f9c411e1196a4360bad1\": rpc error: code = NotFound desc = could not find container \"9c393de72a170f88bd3d33652fd6e9410ee195e2d480f9c411e1196a4360bad1\": container with ID starting with 9c393de72a170f88bd3d33652fd6e9410ee195e2d480f9c411e1196a4360bad1 not found: ID does not exist" Jul 16 21:59:22.246850 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.246825 2577 scope.go:117] "RemoveContainer" containerID="5f92ec3cf32418a4b59c0c62e87e8b01c39a8949d670de2ca047608f49103cad" Jul 16 21:59:22.247050 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:59:22.247031 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f92ec3cf32418a4b59c0c62e87e8b01c39a8949d670de2ca047608f49103cad\": container with ID starting with 5f92ec3cf32418a4b59c0c62e87e8b01c39a8949d670de2ca047608f49103cad not found: ID does not exist" containerID="5f92ec3cf32418a4b59c0c62e87e8b01c39a8949d670de2ca047608f49103cad" Jul 16 21:59:22.247091 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.247056 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f92ec3cf32418a4b59c0c62e87e8b01c39a8949d670de2ca047608f49103cad"} err="failed to get container status \"5f92ec3cf32418a4b59c0c62e87e8b01c39a8949d670de2ca047608f49103cad\": rpc error: code = NotFound desc = could not find container \"5f92ec3cf32418a4b59c0c62e87e8b01c39a8949d670de2ca047608f49103cad\": container with ID starting with 5f92ec3cf32418a4b59c0c62e87e8b01c39a8949d670de2ca047608f49103cad not found: ID does not exist" Jul 16 21:59:22.253288 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.253267 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf"] Jul 16 21:59:22.258530 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.258512 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6e4744-openshift-default-7d75b5b475-2dqdf"] Jul 16 21:59:22.445270 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.445246 2577 scope.go:117] "RemoveContainer" containerID="b0468345851d0ff208951101616c12e6d367d3f1e4df0c6e856a1d5d50d1ce4f" Jul 16 21:59:22.450689 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:22.450670 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a20bca18-8725-49cd-8471-ff4a2f04aa8b" path="/var/lib/kubelet/pods/a20bca18-8725-49cd-8471-ff4a2f04aa8b/volumes" Jul 16 21:59:23.239254 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:23.239225 2577 generic.go:356] "Generic (PLEG): container finished" podID="e88f7139-ef46-468d-b503-efd26517102f" containerID="b760b4318b1c27ed13671417d0ce1b15406a960d506824483ee7619d1d0532f2" exitCode=6 Jul 16 21:59:23.239769 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:23.239306 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" event={"ID":"e88f7139-ef46-468d-b503-efd26517102f","Type":"ContainerDied","Data":"b760b4318b1c27ed13671417d0ce1b15406a960d506824483ee7619d1d0532f2"} Jul 16 21:59:23.239769 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:23.239371 2577 scope.go:117] "RemoveContainer" containerID="b0468345851d0ff208951101616c12e6d367d3f1e4df0c6e856a1d5d50d1ce4f" Jul 16 21:59:23.239769 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:23.239638 2577 scope.go:117] "RemoveContainer" containerID="b760b4318b1c27ed13671417d0ce1b15406a960d506824483ee7619d1d0532f2" Jul 16 21:59:23.239950 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:59:23.239876 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr_odh-ai-gateway-infra(e88f7139-ef46-468d-b503-efd26517102f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" podUID="e88f7139-ef46-468d-b503-efd26517102f" Jul 16 21:59:23.240874 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:23.240843 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" event={"ID":"a6c81b8a-5983-4649-aa15-b93b2bb828a9","Type":"ContainerStarted","Data":"f4a7e75a2867ba94483f8c057f5d774868b7e2552e1b8208043120405d741ec4"} Jul 16 21:59:23.240997 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:23.240920 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" Jul 16 21:59:23.284325 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:23.284282 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" podStartSLOduration=1.637391365 podStartE2EDuration="2.284267771s" totalImagesPullingTime="646.876406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:59:23.282640196 +0000 UTC m=+2979.457748371" watchObservedRunningTime="2026-07-16 21:59:23.284267771 +0000 UTC m=+2979.459375963" Jul 16 21:59:24.271477 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:24.271445 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr"] Jul 16 21:59:24.401451 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:24.401430 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" Jul 16 21:59:24.496044 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:24.496013 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e88f7139-ef46-468d-b503-efd26517102f-maas-api-service-ca\") pod \"e88f7139-ef46-468d-b503-efd26517102f\" (UID: \"e88f7139-ef46-468d-b503-efd26517102f\") " Jul 16 21:59:24.496347 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:24.496326 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e88f7139-ef46-468d-b503-efd26517102f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e88f7139-ef46-468d-b503-efd26517102f" (UID: "e88f7139-ef46-468d-b503-efd26517102f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:59:24.597297 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:24.597230 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e88f7139-ef46-468d-b503-efd26517102f-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:59:25.254567 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:25.254543 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" Jul 16 21:59:25.254735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:25.254548 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr" event={"ID":"e88f7139-ef46-468d-b503-efd26517102f","Type":"ContainerDied","Data":"441cdeffade798ec7d18f12715209f9701bbfda05024846614b6bcf5f4b8b477"} Jul 16 21:59:25.254735 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:25.254661 2577 scope.go:117] "RemoveContainer" containerID="b760b4318b1c27ed13671417d0ce1b15406a960d506824483ee7619d1d0532f2" Jul 16 21:59:25.286237 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:25.286215 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr"] Jul 16 21:59:25.291403 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:25.291380 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-s8pdr"] Jul 16 21:59:26.451634 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.451598 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e88f7139-ef46-468d-b503-efd26517102f" path="/var/lib/kubelet/pods/e88f7139-ef46-468d-b503-efd26517102f/volumes" Jul 16 21:59:26.483969 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.483938 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w"] Jul 16 21:59:26.484296 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.484282 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a20bca18-8725-49cd-8471-ff4a2f04aa8b" containerName="istio-proxy" Jul 16 21:59:26.484296 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.484296 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a20bca18-8725-49cd-8471-ff4a2f04aa8b" containerName="istio-proxy" Jul 16 21:59:26.484429 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.484304 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e88f7139-ef46-468d-b503-efd26517102f" containerName="revoke-keys" Jul 16 21:59:26.484429 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.484311 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e88f7139-ef46-468d-b503-efd26517102f" containerName="revoke-keys" Jul 16 21:59:26.484429 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.484328 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e88f7139-ef46-468d-b503-efd26517102f" containerName="revoke-keys" Jul 16 21:59:26.484429 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.484336 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e88f7139-ef46-468d-b503-efd26517102f" containerName="revoke-keys" Jul 16 21:59:26.484429 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.484347 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e88f7139-ef46-468d-b503-efd26517102f" containerName="revoke-keys" Jul 16 21:59:26.484429 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.484353 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e88f7139-ef46-468d-b503-efd26517102f" containerName="revoke-keys" Jul 16 21:59:26.484429 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.484364 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a20bca18-8725-49cd-8471-ff4a2f04aa8b" containerName="istio-proxy" Jul 16 21:59:26.484429 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.484376 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a20bca18-8725-49cd-8471-ff4a2f04aa8b" containerName="istio-proxy" Jul 16 21:59:26.484720 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.484462 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a20bca18-8725-49cd-8471-ff4a2f04aa8b" containerName="istio-proxy" Jul 16 21:59:26.484720 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.484472 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e88f7139-ef46-468d-b503-efd26517102f" containerName="revoke-keys" Jul 16 21:59:26.484720 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.484479 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a20bca18-8725-49cd-8471-ff4a2f04aa8b" containerName="istio-proxy" Jul 16 21:59:26.484720 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.484489 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e88f7139-ef46-468d-b503-efd26517102f" containerName="revoke-keys" Jul 16 21:59:26.489146 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.489129 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.497465 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.497449 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ef3576-openshift-default-dockercfg-ngvh2\"" Jul 16 21:59:26.511365 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.511345 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w"] Jul 16 21:59:26.610903 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.610870 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-workload-certs\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.611071 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.610911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-istio-data\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.611071 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.610950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-workload-socket\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.611071 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.610970 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492ca68d-51fb-43b0-9707-e6cf13963371-istio-podinfo\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.611071 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.611000 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492ca68d-51fb-43b0-9707-e6cf13963371-istiod-ca-cert\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.611251 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.611091 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92f6z\" (UniqueName: \"kubernetes.io/projected/492ca68d-51fb-43b0-9707-e6cf13963371-kube-api-access-92f6z\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.611251 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.611128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-credential-socket\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.611251 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.611148 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-istio-envoy\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.611251 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.611175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492ca68d-51fb-43b0-9707-e6cf13963371-istio-token\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.711802 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.711732 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-istio-data\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.711802 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.711772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-workload-socket\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.711802 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.711791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492ca68d-51fb-43b0-9707-e6cf13963371-istio-podinfo\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.712070 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.711807 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492ca68d-51fb-43b0-9707-e6cf13963371-istiod-ca-cert\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.712070 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.711842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92f6z\" (UniqueName: \"kubernetes.io/projected/492ca68d-51fb-43b0-9707-e6cf13963371-kube-api-access-92f6z\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.712070 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.711864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-credential-socket\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.712070 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.711883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-istio-envoy\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.712070 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.711897 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492ca68d-51fb-43b0-9707-e6cf13963371-istio-token\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.712070 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.711941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-workload-certs\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.712374 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.712218 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-workload-socket\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.712374 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.712256 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-istio-data\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.712464 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.712367 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-credential-socket\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.712528 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.712468 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-workload-certs\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.712585 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.712558 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492ca68d-51fb-43b0-9707-e6cf13963371-istiod-ca-cert\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.714209 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.714173 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-istio-envoy\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.714288 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.714273 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492ca68d-51fb-43b0-9707-e6cf13963371-istio-podinfo\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.726850 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.726827 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492ca68d-51fb-43b0-9707-e6cf13963371-istio-token\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.728365 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.728334 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92f6z\" (UniqueName: \"kubernetes.io/projected/492ca68d-51fb-43b0-9707-e6cf13963371-kube-api-access-92f6z\") pod \"e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.799122 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.799098 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:26.945229 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.945177 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w"] Jul 16 21:59:26.946995 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:59:26.946961 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492ca68d_51fb_43b0_9707_e6cf13963371.slice/crio-d729bf3e16c4e7cd6fb06e58d5e4f975370ae759f310e283bdcac6ed8ff7889d WatchSource:0}: Error finding container d729bf3e16c4e7cd6fb06e58d5e4f975370ae759f310e283bdcac6ed8ff7889d: Status 404 returned error can't find the container with id d729bf3e16c4e7cd6fb06e58d5e4f975370ae759f310e283bdcac6ed8ff7889d Jul 16 21:59:26.949170 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.949139 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:59:26.949257 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.949227 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:59:26.949301 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:26.949283 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:59:27.269551 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:27.269469 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" event={"ID":"492ca68d-51fb-43b0-9707-e6cf13963371","Type":"ContainerStarted","Data":"53939a027851c7ebb193e45c9273be8c2017701ebd9f830c374c7da43e865765"} Jul 16 21:59:27.269551 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:27.269521 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" event={"ID":"492ca68d-51fb-43b0-9707-e6cf13963371","Type":"ContainerStarted","Data":"d729bf3e16c4e7cd6fb06e58d5e4f975370ae759f310e283bdcac6ed8ff7889d"} Jul 16 21:59:27.297369 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:27.297322 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" podStartSLOduration=1.29730737 podStartE2EDuration="1.29730737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:59:27.293851593 +0000 UTC m=+2983.468959786" watchObservedRunningTime="2026-07-16 21:59:27.29730737 +0000 UTC m=+2983.472415562" Jul 16 21:59:27.799762 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:27.799734 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:28.804741 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:28.804716 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:29.254066 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.254041 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" Jul 16 21:59:29.280922 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.280893 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:29.281873 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.281852 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:29.721434 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.721410 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx"] Jul 16 21:59:29.721803 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.721792 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e88f7139-ef46-468d-b503-efd26517102f" containerName="revoke-keys" Jul 16 21:59:29.724878 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.724864 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" Jul 16 21:59:29.729152 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.729134 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ef3576-serving-cert\"" Jul 16 21:59:29.795309 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.795279 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx"] Jul 16 21:59:29.838040 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.838008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-openshift-service-ca\") pod \"maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx\" (UID: \"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" Jul 16 21:59:29.838430 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.838053 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-maas-api-tls\") pod \"maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx\" (UID: \"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" Jul 16 21:59:29.838430 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.838120 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn6ps\" (UniqueName: \"kubernetes.io/projected/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-kube-api-access-wn6ps\") pod \"maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx\" (UID: \"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" Jul 16 21:59:29.938895 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.938859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-maas-api-tls\") pod \"maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx\" (UID: \"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" Jul 16 21:59:29.939070 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.938925 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn6ps\" (UniqueName: \"kubernetes.io/projected/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-kube-api-access-wn6ps\") pod \"maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx\" (UID: \"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" Jul 16 21:59:29.939070 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.938969 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-openshift-service-ca\") pod \"maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx\" (UID: \"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" Jul 16 21:59:29.939619 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.939598 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-openshift-service-ca\") pod \"maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx\" (UID: \"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" Jul 16 21:59:29.941235 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.941215 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-maas-api-tls\") pod \"maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx\" (UID: \"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" Jul 16 21:59:29.951134 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:29.951082 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn6ps\" (UniqueName: \"kubernetes.io/projected/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-kube-api-access-wn6ps\") pod \"maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx\" (UID: \"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" Jul 16 21:59:30.037050 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:30.037026 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" Jul 16 21:59:30.207138 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:30.207114 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx"] Jul 16 21:59:30.208881 ip-10-0-137-133 kubenswrapper[2577]: W0716 21:59:30.208856 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b88dbe0_f72a_4e6d_98e8_8a590ed19c56.slice/crio-0f0837e0f33fae2877357b0a8f53ac744f2beab8cf4ef790f0c8e5e5dcb9e025 WatchSource:0}: Error finding container 0f0837e0f33fae2877357b0a8f53ac744f2beab8cf4ef790f0c8e5e5dcb9e025: Status 404 returned error can't find the container with id 0f0837e0f33fae2877357b0a8f53ac744f2beab8cf4ef790f0c8e5e5dcb9e025 Jul 16 21:59:30.287090 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:30.287055 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" event={"ID":"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56","Type":"ContainerStarted","Data":"0f0837e0f33fae2877357b0a8f53ac744f2beab8cf4ef790f0c8e5e5dcb9e025"} Jul 16 21:59:31.293316 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:31.293277 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" event={"ID":"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56","Type":"ContainerStarted","Data":"acdab26a1ac75f386be1beef9ed2089658c912e0e3526547633bb573da1e542c"} Jul 16 21:59:31.293782 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:31.293503 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" Jul 16 21:59:31.326032 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:31.325988 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" podStartSLOduration=1.9949351640000001 podStartE2EDuration="2.325974915s" totalImagesPullingTime="331.039751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:59:31.323539516 +0000 UTC m=+2987.498647739" watchObservedRunningTime="2026-07-16 21:59:31.325974915 +0000 UTC m=+2987.501083106" Jul 16 21:59:37.305496 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:37.305470 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" Jul 16 21:59:44.541014 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:44.540989 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:59:44.549017 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:44.548997 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 21:59:49.390923 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:49.390890 2577 generic.go:356] "Generic (PLEG): container finished" podID="492ca68d-51fb-43b0-9707-e6cf13963371" containerID="53939a027851c7ebb193e45c9273be8c2017701ebd9f830c374c7da43e865765" exitCode=0 Jul 16 21:59:49.391466 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:49.390973 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" event={"ID":"492ca68d-51fb-43b0-9707-e6cf13963371","Type":"ContainerDied","Data":"53939a027851c7ebb193e45c9273be8c2017701ebd9f830c374c7da43e865765"} Jul 16 21:59:49.391537 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:49.391503 2577 scope.go:117] "RemoveContainer" containerID="53939a027851c7ebb193e45c9273be8c2017701ebd9f830c374c7da43e865765" Jul 16 21:59:49.392539 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:49.392511 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:59:49.392600 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:49.392578 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:59:49.392637 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:49.392626 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 21:59:50.401972 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:50.401934 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" event={"ID":"492ca68d-51fb-43b0-9707-e6cf13963371","Type":"ContainerStarted","Data":"953f7e54479384aa7d7d337cf0c5224d5714a485d28ab7f45ce93dcb452beef9"} Jul 16 21:59:50.800435 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:50.800331 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:50.800624 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:50.800467 2577 patch_prober.go:28] interesting pod/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Jul 16 21:59:50.800624 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:50.800514 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" podUID="492ca68d-51fb-43b0-9707-e6cf13963371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Jul 16 21:59:51.800033 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:51.799994 2577 patch_prober.go:28] interesting pod/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Jul 16 21:59:51.800664 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:51.800631 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" podUID="492ca68d-51fb-43b0-9707-e6cf13963371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Jul 16 21:59:52.314248 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:52.314215 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq"] Jul 16 21:59:52.314458 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:52.314436 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" podUID="a6c81b8a-5983-4649-aa15-b93b2bb828a9" containerName="maas-api" containerID="cri-o://f4a7e75a2867ba94483f8c057f5d774868b7e2552e1b8208043120405d741ec4" gracePeriod=30 Jul 16 21:59:52.606274 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:52.606241 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" Jul 16 21:59:52.742907 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:52.742866 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c81b8a-5983-4649-aa15-b93b2bb828a9-maas-api-tls\") pod \"a6c81b8a-5983-4649-aa15-b93b2bb828a9\" (UID: \"a6c81b8a-5983-4649-aa15-b93b2bb828a9\") " Jul 16 21:59:52.743109 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:52.742977 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hvmh\" (UniqueName: \"kubernetes.io/projected/a6c81b8a-5983-4649-aa15-b93b2bb828a9-kube-api-access-4hvmh\") pod \"a6c81b8a-5983-4649-aa15-b93b2bb828a9\" (UID: \"a6c81b8a-5983-4649-aa15-b93b2bb828a9\") " Jul 16 21:59:52.743109 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:52.743007 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c81b8a-5983-4649-aa15-b93b2bb828a9-openshift-service-ca\") pod \"a6c81b8a-5983-4649-aa15-b93b2bb828a9\" (UID: \"a6c81b8a-5983-4649-aa15-b93b2bb828a9\") " Jul 16 21:59:52.743408 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:52.743381 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c81b8a-5983-4649-aa15-b93b2bb828a9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a6c81b8a-5983-4649-aa15-b93b2bb828a9" (UID: "a6c81b8a-5983-4649-aa15-b93b2bb828a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:59:52.744934 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:52.744907 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6c81b8a-5983-4649-aa15-b93b2bb828a9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a6c81b8a-5983-4649-aa15-b93b2bb828a9" (UID: "a6c81b8a-5983-4649-aa15-b93b2bb828a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:59:52.745059 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:52.745012 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c81b8a-5983-4649-aa15-b93b2bb828a9-kube-api-access-4hvmh" (OuterVolumeSpecName: "kube-api-access-4hvmh") pod "a6c81b8a-5983-4649-aa15-b93b2bb828a9" (UID: "a6c81b8a-5983-4649-aa15-b93b2bb828a9"). InnerVolumeSpecName "kube-api-access-4hvmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:59:52.800613 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:52.800148 2577 patch_prober.go:28] interesting pod/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Jul 16 21:59:52.800613 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:52.800222 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" podUID="492ca68d-51fb-43b0-9707-e6cf13963371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Jul 16 21:59:52.844016 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:52.843954 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hvmh\" (UniqueName: \"kubernetes.io/projected/a6c81b8a-5983-4649-aa15-b93b2bb828a9-kube-api-access-4hvmh\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:59:52.844016 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:52.843976 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c81b8a-5983-4649-aa15-b93b2bb828a9-openshift-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:59:52.844016 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:52.843986 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c81b8a-5983-4649-aa15-b93b2bb828a9-maas-api-tls\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 21:59:53.418474 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:53.418436 2577 generic.go:356] "Generic (PLEG): container finished" podID="a6c81b8a-5983-4649-aa15-b93b2bb828a9" containerID="f4a7e75a2867ba94483f8c057f5d774868b7e2552e1b8208043120405d741ec4" exitCode=0 Jul 16 21:59:53.418678 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:53.418481 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" Jul 16 21:59:53.418678 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:53.418519 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" event={"ID":"a6c81b8a-5983-4649-aa15-b93b2bb828a9","Type":"ContainerDied","Data":"f4a7e75a2867ba94483f8c057f5d774868b7e2552e1b8208043120405d741ec4"} Jul 16 21:59:53.418678 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:53.418574 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq" event={"ID":"a6c81b8a-5983-4649-aa15-b93b2bb828a9","Type":"ContainerDied","Data":"6d40b3cf4690159e921a0f94e2576d5af59bfd7a5a753f2ebfd03fc2e641bb54"} Jul 16 21:59:53.418678 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:53.418598 2577 scope.go:117] "RemoveContainer" containerID="f4a7e75a2867ba94483f8c057f5d774868b7e2552e1b8208043120405d741ec4" Jul 16 21:59:53.427674 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:53.427659 2577 scope.go:117] "RemoveContainer" containerID="f4a7e75a2867ba94483f8c057f5d774868b7e2552e1b8208043120405d741ec4" Jul 16 21:59:53.427916 ip-10-0-137-133 kubenswrapper[2577]: E0716 21:59:53.427899 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a7e75a2867ba94483f8c057f5d774868b7e2552e1b8208043120405d741ec4\": container with ID starting with f4a7e75a2867ba94483f8c057f5d774868b7e2552e1b8208043120405d741ec4 not found: ID does not exist" containerID="f4a7e75a2867ba94483f8c057f5d774868b7e2552e1b8208043120405d741ec4" Jul 16 21:59:53.427968 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:53.427924 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a7e75a2867ba94483f8c057f5d774868b7e2552e1b8208043120405d741ec4"} err="failed to get container status \"f4a7e75a2867ba94483f8c057f5d774868b7e2552e1b8208043120405d741ec4\": rpc error: code = NotFound desc = could not find container \"f4a7e75a2867ba94483f8c057f5d774868b7e2552e1b8208043120405d741ec4\": container with ID starting with f4a7e75a2867ba94483f8c057f5d774868b7e2552e1b8208043120405d741ec4 not found: ID does not exist" Jul 16 21:59:53.441937 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:53.441916 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq"] Jul 16 21:59:53.446471 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:53.446451 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f90d76-6c857c69d7-2skqq"] Jul 16 21:59:53.800278 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:53.800158 2577 patch_prober.go:28] interesting pod/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Jul 16 21:59:53.800571 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:53.800531 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" podUID="492ca68d-51fb-43b0-9707-e6cf13963371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Jul 16 21:59:54.450150 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:54.450109 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6c81b8a-5983-4649-aa15-b93b2bb828a9" path="/var/lib/kubelet/pods/a6c81b8a-5983-4649-aa15-b93b2bb828a9/volumes" Jul 16 21:59:54.803996 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:54.803927 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:54.804224 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:54.804206 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 21:59:54.804793 ip-10-0-137-133 kubenswrapper[2577]: I0716 21:59:54.804776 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 22:00:00.134338 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:00.134303 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737320-dz2bj"] Jul 16 22:00:00.134829 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:00.134663 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6c81b8a-5983-4649-aa15-b93b2bb828a9" containerName="maas-api" Jul 16 22:00:00.134829 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:00.134674 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6c81b8a-5983-4649-aa15-b93b2bb828a9" containerName="maas-api" Jul 16 22:00:00.134829 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:00.134729 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6c81b8a-5983-4649-aa15-b93b2bb828a9" containerName="maas-api" Jul 16 22:00:00.137620 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:00.137604 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-dz2bj" Jul 16 22:00:00.140509 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:00.140485 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 22:00:00.147110 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:00.147086 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737320-dz2bj"] Jul 16 22:00:00.148791 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:00.148771 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-dz2bj" Jul 16 22:00:00.490267 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:00.490244 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737320-dz2bj"] Jul 16 22:00:00.492088 ip-10-0-137-133 kubenswrapper[2577]: W0716 22:00:00.492059 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62dc075_1101_4952_a656_339b5d148750.slice/crio-e91af1b0b0be6d928bc68f5b7e3c8195bfcb8a098b924b75df39e58c3f27e640 WatchSource:0}: Error finding container e91af1b0b0be6d928bc68f5b7e3c8195bfcb8a098b924b75df39e58c3f27e640: Status 404 returned error can't find the container with id e91af1b0b0be6d928bc68f5b7e3c8195bfcb8a098b924b75df39e58c3f27e640 Jul 16 22:00:01.469824 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:01.469790 2577 generic.go:356] "Generic (PLEG): container finished" podID="f62dc075-1101-4952-a656-339b5d148750" containerID="9077d95fcc0ba07c6c83927747d0c750a8f86fde4217dc1587fa0d3a1eabc6ad" exitCode=0 Jul 16 22:00:01.470251 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:01.469851 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-dz2bj" event={"ID":"f62dc075-1101-4952-a656-339b5d148750","Type":"ContainerDied","Data":"9077d95fcc0ba07c6c83927747d0c750a8f86fde4217dc1587fa0d3a1eabc6ad"} Jul 16 22:00:01.470251 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:01.469903 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-dz2bj" event={"ID":"f62dc075-1101-4952-a656-339b5d148750","Type":"ContainerStarted","Data":"e91af1b0b0be6d928bc68f5b7e3c8195bfcb8a098b924b75df39e58c3f27e640"} Jul 16 22:00:02.606864 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:02.606841 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-dz2bj" Jul 16 22:00:03.481930 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:03.481896 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-dz2bj" Jul 16 22:00:03.482104 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:03.481934 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-dz2bj" event={"ID":"f62dc075-1101-4952-a656-339b5d148750","Type":"ContainerDied","Data":"e91af1b0b0be6d928bc68f5b7e3c8195bfcb8a098b924b75df39e58c3f27e640"} Jul 16 22:00:03.482104 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:03.481987 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e91af1b0b0be6d928bc68f5b7e3c8195bfcb8a098b924b75df39e58c3f27e640" Jul 16 22:00:19.221407 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.221375 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx"] Jul 16 22:00:19.221979 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.221611 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" podUID="0b88dbe0-f72a-4e6d-98e8-8a590ed19c56" containerName="maas-api" containerID="cri-o://acdab26a1ac75f386be1beef9ed2089658c912e0e3526547633bb573da1e542c" gracePeriod=30 Jul 16 22:00:19.463536 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.463506 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" Jul 16 22:00:19.572693 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.572612 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-openshift-service-ca\") pod \"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56\" (UID: \"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56\") " Jul 16 22:00:19.572693 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.572658 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-maas-api-tls\") pod \"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56\" (UID: \"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56\") " Jul 16 22:00:19.572909 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.572697 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn6ps\" (UniqueName: \"kubernetes.io/projected/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-kube-api-access-wn6ps\") pod \"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56\" (UID: \"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56\") " Jul 16 22:00:19.572909 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.572777 2577 generic.go:356] "Generic (PLEG): container finished" podID="0b88dbe0-f72a-4e6d-98e8-8a590ed19c56" containerID="acdab26a1ac75f386be1beef9ed2089658c912e0e3526547633bb573da1e542c" exitCode=0 Jul 16 22:00:19.572909 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.572850 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" Jul 16 22:00:19.573023 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.572911 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" event={"ID":"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56","Type":"ContainerDied","Data":"acdab26a1ac75f386be1beef9ed2089658c912e0e3526547633bb573da1e542c"} Jul 16 22:00:19.573023 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.572981 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx" event={"ID":"0b88dbe0-f72a-4e6d-98e8-8a590ed19c56","Type":"ContainerDied","Data":"0f0837e0f33fae2877357b0a8f53ac744f2beab8cf4ef790f0c8e5e5dcb9e025"} Jul 16 22:00:19.573023 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.573014 2577 scope.go:117] "RemoveContainer" containerID="acdab26a1ac75f386be1beef9ed2089658c912e0e3526547633bb573da1e542c" Jul 16 22:00:19.573023 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.573012 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0b88dbe0-f72a-4e6d-98e8-8a590ed19c56" (UID: "0b88dbe0-f72a-4e6d-98e8-8a590ed19c56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:00:19.575277 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.575255 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0b88dbe0-f72a-4e6d-98e8-8a590ed19c56" (UID: "0b88dbe0-f72a-4e6d-98e8-8a590ed19c56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:00:19.575277 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.575252 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-kube-api-access-wn6ps" (OuterVolumeSpecName: "kube-api-access-wn6ps") pod "0b88dbe0-f72a-4e6d-98e8-8a590ed19c56" (UID: "0b88dbe0-f72a-4e6d-98e8-8a590ed19c56"). InnerVolumeSpecName "kube-api-access-wn6ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:00:19.590953 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.590938 2577 scope.go:117] "RemoveContainer" containerID="acdab26a1ac75f386be1beef9ed2089658c912e0e3526547633bb573da1e542c" Jul 16 22:00:19.591253 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:00:19.591235 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acdab26a1ac75f386be1beef9ed2089658c912e0e3526547633bb573da1e542c\": container with ID starting with acdab26a1ac75f386be1beef9ed2089658c912e0e3526547633bb573da1e542c not found: ID does not exist" containerID="acdab26a1ac75f386be1beef9ed2089658c912e0e3526547633bb573da1e542c" Jul 16 22:00:19.591317 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.591262 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acdab26a1ac75f386be1beef9ed2089658c912e0e3526547633bb573da1e542c"} err="failed to get container status \"acdab26a1ac75f386be1beef9ed2089658c912e0e3526547633bb573da1e542c\": rpc error: code = NotFound desc = could not find container \"acdab26a1ac75f386be1beef9ed2089658c912e0e3526547633bb573da1e542c\": container with ID starting with acdab26a1ac75f386be1beef9ed2089658c912e0e3526547633bb573da1e542c not found: ID does not exist" Jul 16 22:00:19.673898 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.673874 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn6ps\" (UniqueName: \"kubernetes.io/projected/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-kube-api-access-wn6ps\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:00:19.673898 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.673895 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-openshift-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:00:19.674025 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.673906 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56-maas-api-tls\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:00:19.896270 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.896241 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx"] Jul 16 22:00:19.900636 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:19.900617 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ef3576-85548b7b4b-nqvxx"] Jul 16 22:00:20.450719 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:20.450688 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b88dbe0-f72a-4e6d-98e8-8a590ed19c56" path="/var/lib/kubelet/pods/0b88dbe0-f72a-4e6d-98e8-8a590ed19c56/volumes" Jul 16 22:00:34.036034 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:34.035999 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w"] Jul 16 22:00:34.036580 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:34.036218 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" podUID="492ca68d-51fb-43b0-9707-e6cf13963371" containerName="istio-proxy" containerID="cri-o://953f7e54479384aa7d7d337cf0c5224d5714a485d28ab7f45ce93dcb452beef9" gracePeriod=30 Jul 16 22:00:37.624475 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.624394 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj"] Jul 16 22:00:37.624988 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.624901 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b88dbe0-f72a-4e6d-98e8-8a590ed19c56" containerName="maas-api" Jul 16 22:00:37.624988 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.624920 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b88dbe0-f72a-4e6d-98e8-8a590ed19c56" containerName="maas-api" Jul 16 22:00:37.624988 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.624938 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f62dc075-1101-4952-a656-339b5d148750" containerName="cleanup" Jul 16 22:00:37.624988 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.624946 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f62dc075-1101-4952-a656-339b5d148750" containerName="cleanup" Jul 16 22:00:37.625225 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.625017 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f62dc075-1101-4952-a656-339b5d148750" containerName="cleanup" Jul 16 22:00:37.625225 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.625034 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b88dbe0-f72a-4e6d-98e8-8a590ed19c56" containerName="maas-api" Jul 16 22:00:37.628135 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.628115 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" Jul 16 22:00:37.631962 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.631944 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xcg5c\"" Jul 16 22:00:37.632541 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.632521 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-2140ea-serving-cert\"" Jul 16 22:00:37.632649 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.632553 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 22:00:37.632875 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.632861 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 22:00:37.639962 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.639945 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj"] Jul 16 22:00:37.715515 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.715491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj\" (UID: \"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" Jul 16 22:00:37.715648 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.715532 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-maas-api-tls\") pod \"maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj\" (UID: \"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" Jul 16 22:00:37.715648 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.715589 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77kmd\" (UniqueName: \"kubernetes.io/projected/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-kube-api-access-77kmd\") pod \"maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj\" (UID: \"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" Jul 16 22:00:37.816223 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.816169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj\" (UID: \"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" Jul 16 22:00:37.816555 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.816490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-maas-api-tls\") pod \"maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj\" (UID: \"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" Jul 16 22:00:37.816684 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.816565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77kmd\" (UniqueName: \"kubernetes.io/projected/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-kube-api-access-77kmd\") pod \"maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj\" (UID: \"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" Jul 16 22:00:37.817081 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.817055 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj\" (UID: \"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" Jul 16 22:00:37.820358 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.820336 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-maas-api-tls\") pod \"maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj\" (UID: \"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" Jul 16 22:00:37.831170 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.831148 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77kmd\" (UniqueName: \"kubernetes.io/projected/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-kube-api-access-77kmd\") pod \"maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj\" (UID: \"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" Jul 16 22:00:37.938420 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:37.938397 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" Jul 16 22:00:38.073956 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:38.073927 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj"] Jul 16 22:00:38.078886 ip-10-0-137-133 kubenswrapper[2577]: W0716 22:00:38.078836 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1561f4_851b_46a1_bd8c_1778d5ef8b0f.slice/crio-33ca6b0d758936faffbcad5ae1d858321bb7ff0b6bf3774e3abc5762d46af90e WatchSource:0}: Error finding container 33ca6b0d758936faffbcad5ae1d858321bb7ff0b6bf3774e3abc5762d46af90e: Status 404 returned error can't find the container with id 33ca6b0d758936faffbcad5ae1d858321bb7ff0b6bf3774e3abc5762d46af90e Jul 16 22:00:38.674966 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:38.674930 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" event={"ID":"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f","Type":"ContainerStarted","Data":"33ca6b0d758936faffbcad5ae1d858321bb7ff0b6bf3774e3abc5762d46af90e"} Jul 16 22:00:39.280001 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.279981 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 22:00:39.431277 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.431252 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-workload-certs\") pod \"492ca68d-51fb-43b0-9707-e6cf13963371\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " Jul 16 22:00:39.431277 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.431279 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-istio-envoy\") pod \"492ca68d-51fb-43b0-9707-e6cf13963371\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " Jul 16 22:00:39.431483 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.431298 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492ca68d-51fb-43b0-9707-e6cf13963371-istiod-ca-cert\") pod \"492ca68d-51fb-43b0-9707-e6cf13963371\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " Jul 16 22:00:39.431483 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.431365 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492ca68d-51fb-43b0-9707-e6cf13963371-istio-token\") pod \"492ca68d-51fb-43b0-9707-e6cf13963371\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " Jul 16 22:00:39.431483 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.431401 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-workload-socket\") pod \"492ca68d-51fb-43b0-9707-e6cf13963371\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " Jul 16 22:00:39.431483 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.431426 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492ca68d-51fb-43b0-9707-e6cf13963371-istio-podinfo\") pod \"492ca68d-51fb-43b0-9707-e6cf13963371\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " Jul 16 22:00:39.431684 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.431496 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92f6z\" (UniqueName: \"kubernetes.io/projected/492ca68d-51fb-43b0-9707-e6cf13963371-kube-api-access-92f6z\") pod \"492ca68d-51fb-43b0-9707-e6cf13963371\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " Jul 16 22:00:39.431684 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.431536 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "492ca68d-51fb-43b0-9707-e6cf13963371" (UID: "492ca68d-51fb-43b0-9707-e6cf13963371"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:00:39.431684 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.431626 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-credential-socket\") pod \"492ca68d-51fb-43b0-9707-e6cf13963371\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " Jul 16 22:00:39.431684 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.431660 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-istio-data\") pod \"492ca68d-51fb-43b0-9707-e6cf13963371\" (UID: \"492ca68d-51fb-43b0-9707-e6cf13963371\") " Jul 16 22:00:39.431898 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.431706 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492ca68d-51fb-43b0-9707-e6cf13963371-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "492ca68d-51fb-43b0-9707-e6cf13963371" (UID: "492ca68d-51fb-43b0-9707-e6cf13963371"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:00:39.431898 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.431810 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "492ca68d-51fb-43b0-9707-e6cf13963371" (UID: "492ca68d-51fb-43b0-9707-e6cf13963371"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:00:39.432007 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.431970 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-workload-socket\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:00:39.432007 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.431987 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-workload-certs\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:00:39.432007 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.431994 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "492ca68d-51fb-43b0-9707-e6cf13963371" (UID: "492ca68d-51fb-43b0-9707-e6cf13963371"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:00:39.432156 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.432001 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492ca68d-51fb-43b0-9707-e6cf13963371-istiod-ca-cert\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:00:39.432629 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.432602 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-istio-data" (OuterVolumeSpecName: "istio-data") pod "492ca68d-51fb-43b0-9707-e6cf13963371" (UID: "492ca68d-51fb-43b0-9707-e6cf13963371"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:00:39.433458 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.433438 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "492ca68d-51fb-43b0-9707-e6cf13963371" (UID: "492ca68d-51fb-43b0-9707-e6cf13963371"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:00:39.433644 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.433625 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492ca68d-51fb-43b0-9707-e6cf13963371-istio-token" (OuterVolumeSpecName: "istio-token") pod "492ca68d-51fb-43b0-9707-e6cf13963371" (UID: "492ca68d-51fb-43b0-9707-e6cf13963371"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:00:39.433708 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.433640 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492ca68d-51fb-43b0-9707-e6cf13963371-kube-api-access-92f6z" (OuterVolumeSpecName: "kube-api-access-92f6z") pod "492ca68d-51fb-43b0-9707-e6cf13963371" (UID: "492ca68d-51fb-43b0-9707-e6cf13963371"). InnerVolumeSpecName "kube-api-access-92f6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:00:39.433708 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.433647 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/492ca68d-51fb-43b0-9707-e6cf13963371-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "492ca68d-51fb-43b0-9707-e6cf13963371" (UID: "492ca68d-51fb-43b0-9707-e6cf13963371"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 22:00:39.532388 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.532366 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492ca68d-51fb-43b0-9707-e6cf13963371-istio-podinfo\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:00:39.532388 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.532384 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92f6z\" (UniqueName: \"kubernetes.io/projected/492ca68d-51fb-43b0-9707-e6cf13963371-kube-api-access-92f6z\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:00:39.532561 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.532393 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-credential-socket\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:00:39.532561 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.532402 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-istio-data\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:00:39.532561 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.532410 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492ca68d-51fb-43b0-9707-e6cf13963371-istio-envoy\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:00:39.532561 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.532417 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492ca68d-51fb-43b0-9707-e6cf13963371-istio-token\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:00:39.681612 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.681546 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" event={"ID":"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f","Type":"ContainerStarted","Data":"79e43c2e04d6db0983c27810524ffad217681f39416b11a2ef0d04342e0acadb"} Jul 16 22:00:39.681612 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.681604 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" Jul 16 22:00:39.683082 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.683055 2577 generic.go:356] "Generic (PLEG): container finished" podID="492ca68d-51fb-43b0-9707-e6cf13963371" containerID="953f7e54479384aa7d7d337cf0c5224d5714a485d28ab7f45ce93dcb452beef9" exitCode=0 Jul 16 22:00:39.683202 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.683107 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" event={"ID":"492ca68d-51fb-43b0-9707-e6cf13963371","Type":"ContainerDied","Data":"953f7e54479384aa7d7d337cf0c5224d5714a485d28ab7f45ce93dcb452beef9"} Jul 16 22:00:39.683202 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.683111 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" Jul 16 22:00:39.683202 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.683149 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w" event={"ID":"492ca68d-51fb-43b0-9707-e6cf13963371","Type":"ContainerDied","Data":"d729bf3e16c4e7cd6fb06e58d5e4f975370ae759f310e283bdcac6ed8ff7889d"} Jul 16 22:00:39.683202 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.683171 2577 scope.go:117] "RemoveContainer" containerID="953f7e54479384aa7d7d337cf0c5224d5714a485d28ab7f45ce93dcb452beef9" Jul 16 22:00:39.691659 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.691643 2577 scope.go:117] "RemoveContainer" containerID="53939a027851c7ebb193e45c9273be8c2017701ebd9f830c374c7da43e865765" Jul 16 22:00:39.699495 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.699476 2577 scope.go:117] "RemoveContainer" containerID="953f7e54479384aa7d7d337cf0c5224d5714a485d28ab7f45ce93dcb452beef9" Jul 16 22:00:39.699728 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:00:39.699709 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"953f7e54479384aa7d7d337cf0c5224d5714a485d28ab7f45ce93dcb452beef9\": container with ID starting with 953f7e54479384aa7d7d337cf0c5224d5714a485d28ab7f45ce93dcb452beef9 not found: ID does not exist" containerID="953f7e54479384aa7d7d337cf0c5224d5714a485d28ab7f45ce93dcb452beef9" Jul 16 22:00:39.699786 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.699734 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"953f7e54479384aa7d7d337cf0c5224d5714a485d28ab7f45ce93dcb452beef9"} err="failed to get container status \"953f7e54479384aa7d7d337cf0c5224d5714a485d28ab7f45ce93dcb452beef9\": rpc error: code = NotFound desc = could not find container \"953f7e54479384aa7d7d337cf0c5224d5714a485d28ab7f45ce93dcb452beef9\": container with ID starting with 953f7e54479384aa7d7d337cf0c5224d5714a485d28ab7f45ce93dcb452beef9 not found: ID does not exist" Jul 16 22:00:39.699786 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.699750 2577 scope.go:117] "RemoveContainer" containerID="53939a027851c7ebb193e45c9273be8c2017701ebd9f830c374c7da43e865765" Jul 16 22:00:39.699954 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:00:39.699940 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53939a027851c7ebb193e45c9273be8c2017701ebd9f830c374c7da43e865765\": container with ID starting with 53939a027851c7ebb193e45c9273be8c2017701ebd9f830c374c7da43e865765 not found: ID does not exist" containerID="53939a027851c7ebb193e45c9273be8c2017701ebd9f830c374c7da43e865765" Jul 16 22:00:39.699990 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.699958 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53939a027851c7ebb193e45c9273be8c2017701ebd9f830c374c7da43e865765"} err="failed to get container status \"53939a027851c7ebb193e45c9273be8c2017701ebd9f830c374c7da43e865765\": rpc error: code = NotFound desc = could not find container \"53939a027851c7ebb193e45c9273be8c2017701ebd9f830c374c7da43e865765\": container with ID starting with 53939a027851c7ebb193e45c9273be8c2017701ebd9f830c374c7da43e865765 not found: ID does not exist" Jul 16 22:00:39.704058 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.704024 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" podStartSLOduration=2.093379074 podStartE2EDuration="2.70401357s" totalImagesPullingTime="610.634496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:00:39.701843447 +0000 UTC m=+3055.876951638" watchObservedRunningTime="2026-07-16 22:00:39.70401357 +0000 UTC m=+3055.879121771" Jul 16 22:00:39.723244 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.723223 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w"] Jul 16 22:00:39.729846 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:39.729828 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ef3576-openshift-default-95c6756cc-npn9w"] Jul 16 22:00:40.451173 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:40.451140 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="492ca68d-51fb-43b0-9707-e6cf13963371" path="/var/lib/kubelet/pods/492ca68d-51fb-43b0-9707-e6cf13963371/volumes" Jul 16 22:00:42.506120 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.506090 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg"] Jul 16 22:00:42.506488 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.506444 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492ca68d-51fb-43b0-9707-e6cf13963371" containerName="istio-proxy" Jul 16 22:00:42.506488 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.506456 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="492ca68d-51fb-43b0-9707-e6cf13963371" containerName="istio-proxy" Jul 16 22:00:42.506488 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.506465 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492ca68d-51fb-43b0-9707-e6cf13963371" containerName="istio-proxy" Jul 16 22:00:42.506488 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.506471 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="492ca68d-51fb-43b0-9707-e6cf13963371" containerName="istio-proxy" Jul 16 22:00:42.506652 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.506536 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="492ca68d-51fb-43b0-9707-e6cf13963371" containerName="istio-proxy" Jul 16 22:00:42.506652 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.506545 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="492ca68d-51fb-43b0-9707-e6cf13963371" containerName="istio-proxy" Jul 16 22:00:42.510934 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.510915 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.514007 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.513991 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-a87ce2-openshift-default-dockercfg-rkmvz\"" Jul 16 22:00:42.522337 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.522316 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg"] Jul 16 22:00:42.656155 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.656125 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxx4m\" (UniqueName: \"kubernetes.io/projected/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-kube-api-access-jxx4m\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.656354 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.656161 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-data\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.656354 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.656234 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-envoy\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.656354 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.656287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-podinfo\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.656354 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.656331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istiod-ca-cert\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.656354 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.656348 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-workload-socket\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.656525 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.656421 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-token\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.656525 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.656450 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-workload-certs\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.656525 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.656469 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-credential-socket\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.757568 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.757501 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-podinfo\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.757568 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.757535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istiod-ca-cert\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.757568 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.757556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-workload-socket\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.757778 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.757745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-token\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.757837 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.757792 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-workload-certs\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.757837 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.757821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-credential-socket\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.757932 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.757877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxx4m\" (UniqueName: \"kubernetes.io/projected/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-kube-api-access-jxx4m\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.757932 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.757908 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-data\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.758031 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.757944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-envoy\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.758031 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.757968 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-workload-socket\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.758162 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.758136 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-workload-certs\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.758266 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.758238 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-credential-socket\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.758329 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.758296 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istiod-ca-cert\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.758381 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.758339 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-data\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.760020 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.759994 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-podinfo\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.760123 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.760107 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-envoy\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.780389 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.780359 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-token\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.780499 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.780396 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxx4m\" (UniqueName: \"kubernetes.io/projected/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-kube-api-access-jxx4m\") pod \"e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.821722 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.821697 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:42.954986 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.954949 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg"] Jul 16 22:00:42.955955 ip-10-0-137-133 kubenswrapper[2577]: W0716 22:00:42.955932 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9dbd76_938f_4670_9d14_1f1f3b44ff85.slice/crio-0fbec1b9c17d6ebd31259a796b9f14cd5a733ce4b1d43cd84e83a378de92b38d WatchSource:0}: Error finding container 0fbec1b9c17d6ebd31259a796b9f14cd5a733ce4b1d43cd84e83a378de92b38d: Status 404 returned error can't find the container with id 0fbec1b9c17d6ebd31259a796b9f14cd5a733ce4b1d43cd84e83a378de92b38d Jul 16 22:00:42.957927 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.957897 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 22:00:42.958010 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.957958 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 22:00:42.958010 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:42.957988 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 22:00:43.711640 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:43.711604 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" event={"ID":"bf9dbd76-938f-4670-9d14-1f1f3b44ff85","Type":"ContainerStarted","Data":"be8413ed3c47c5a7569a7575a8180949e67d8392ba2f4e16d4eb8c9a83e9c4cf"} Jul 16 22:00:43.712039 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:43.711657 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" event={"ID":"bf9dbd76-938f-4670-9d14-1f1f3b44ff85","Type":"ContainerStarted","Data":"0fbec1b9c17d6ebd31259a796b9f14cd5a733ce4b1d43cd84e83a378de92b38d"} Jul 16 22:00:43.736744 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:43.736698 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" podStartSLOduration=1.736683439 podStartE2EDuration="1.736683439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:00:43.735351297 +0000 UTC m=+3059.910459490" watchObservedRunningTime="2026-07-16 22:00:43.736683439 +0000 UTC m=+3059.911791629" Jul 16 22:00:43.822649 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:43.822625 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:44.827486 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:44.827456 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:45.701587 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:45.701563 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" Jul 16 22:00:45.721999 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:45.721974 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:45.722914 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:45.722891 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:45.867982 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:45.867949 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8"] Jul 16 22:00:45.872770 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:45.872749 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" Jul 16 22:00:45.875545 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:45.875522 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-a87ce2-serving-cert\"" Jul 16 22:00:45.885016 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:45.884995 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8"] Jul 16 22:00:45.984478 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:45.984386 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c33a47-aba0-456d-a973-9ef9c5471360-maas-api-tls\") pod \"maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8\" (UID: \"b5c33a47-aba0-456d-a973-9ef9c5471360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" Jul 16 22:00:45.984478 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:45.984454 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b6bt\" (UniqueName: \"kubernetes.io/projected/b5c33a47-aba0-456d-a973-9ef9c5471360-kube-api-access-8b6bt\") pod \"maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8\" (UID: \"b5c33a47-aba0-456d-a973-9ef9c5471360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" Jul 16 22:00:45.984709 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:45.984505 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c33a47-aba0-456d-a973-9ef9c5471360-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8\" (UID: \"b5c33a47-aba0-456d-a973-9ef9c5471360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" Jul 16 22:00:46.085697 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:46.085663 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c33a47-aba0-456d-a973-9ef9c5471360-maas-api-tls\") pod \"maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8\" (UID: \"b5c33a47-aba0-456d-a973-9ef9c5471360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" Jul 16 22:00:46.085840 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:46.085718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b6bt\" (UniqueName: \"kubernetes.io/projected/b5c33a47-aba0-456d-a973-9ef9c5471360-kube-api-access-8b6bt\") pod \"maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8\" (UID: \"b5c33a47-aba0-456d-a973-9ef9c5471360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" Jul 16 22:00:46.085896 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:46.085877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c33a47-aba0-456d-a973-9ef9c5471360-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8\" (UID: \"b5c33a47-aba0-456d-a973-9ef9c5471360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" Jul 16 22:00:46.086526 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:46.086494 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c33a47-aba0-456d-a973-9ef9c5471360-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8\" (UID: \"b5c33a47-aba0-456d-a973-9ef9c5471360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" Jul 16 22:00:46.088055 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:46.088037 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c33a47-aba0-456d-a973-9ef9c5471360-maas-api-tls\") pod \"maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8\" (UID: \"b5c33a47-aba0-456d-a973-9ef9c5471360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" Jul 16 22:00:46.095588 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:46.095561 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b6bt\" (UniqueName: \"kubernetes.io/projected/b5c33a47-aba0-456d-a973-9ef9c5471360-kube-api-access-8b6bt\") pod \"maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8\" (UID: \"b5c33a47-aba0-456d-a973-9ef9c5471360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" Jul 16 22:00:46.183646 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:46.183621 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" Jul 16 22:00:46.318000 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:46.317973 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8"] Jul 16 22:00:46.319225 ip-10-0-137-133 kubenswrapper[2577]: W0716 22:00:46.319168 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c33a47_aba0_456d_a973_9ef9c5471360.slice/crio-03bcb526359fa8d2ea92de6b7c1ce2f21d48326044313a9638e71d59e8504758 WatchSource:0}: Error finding container 03bcb526359fa8d2ea92de6b7c1ce2f21d48326044313a9638e71d59e8504758: Status 404 returned error can't find the container with id 03bcb526359fa8d2ea92de6b7c1ce2f21d48326044313a9638e71d59e8504758 Jul 16 22:00:46.727738 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:46.727706 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" event={"ID":"b5c33a47-aba0-456d-a973-9ef9c5471360","Type":"ContainerStarted","Data":"03bcb526359fa8d2ea92de6b7c1ce2f21d48326044313a9638e71d59e8504758"} Jul 16 22:00:47.735343 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:47.735314 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" event={"ID":"b5c33a47-aba0-456d-a973-9ef9c5471360","Type":"ContainerStarted","Data":"e1635d9da59d69dcb93f4756f708009424bdb339db3468c97d41723083cb5ef3"} Jul 16 22:00:47.735697 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:47.735485 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" Jul 16 22:00:47.754795 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:47.754751 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" podStartSLOduration=2.04762615 podStartE2EDuration="2.754737896s" totalImagesPullingTime="707.111746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:00:47.75328258 +0000 UTC m=+3063.928390796" watchObservedRunningTime="2026-07-16 22:00:47.754737896 +0000 UTC m=+3063.929846088" Jul 16 22:00:53.745789 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:53.745760 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" Jul 16 22:00:58.796996 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:58.796963 2577 generic.go:356] "Generic (PLEG): container finished" podID="bf9dbd76-938f-4670-9d14-1f1f3b44ff85" containerID="be8413ed3c47c5a7569a7575a8180949e67d8392ba2f4e16d4eb8c9a83e9c4cf" exitCode=0 Jul 16 22:00:58.797537 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:58.797043 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" event={"ID":"bf9dbd76-938f-4670-9d14-1f1f3b44ff85","Type":"ContainerDied","Data":"be8413ed3c47c5a7569a7575a8180949e67d8392ba2f4e16d4eb8c9a83e9c4cf"} Jul 16 22:00:58.797619 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:58.797536 2577 scope.go:117] "RemoveContainer" containerID="be8413ed3c47c5a7569a7575a8180949e67d8392ba2f4e16d4eb8c9a83e9c4cf" Jul 16 22:00:58.798590 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:58.798560 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 22:00:58.798697 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:58.798624 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 22:00:58.798697 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:58.798656 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 22:00:59.805702 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:59.805662 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" event={"ID":"bf9dbd76-938f-4670-9d14-1f1f3b44ff85","Type":"ContainerStarted","Data":"77bdd743395fd88b78395b71028eef5de051ace203e7532c161edf67fb90e5bc"} Jul 16 22:00:59.822327 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:59.822296 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:00:59.823418 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:59.823396 2577 patch_prober.go:28] interesting pod/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Jul 16 22:00:59.823525 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:00:59.823445 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" podUID="bf9dbd76-938f-4670-9d14-1f1f3b44ff85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Jul 16 22:01:00.722355 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:00.722325 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:01:00.822593 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:00.822561 2577 patch_prober.go:28] interesting pod/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Jul 16 22:01:00.822988 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:00.822613 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" podUID="bf9dbd76-938f-4670-9d14-1f1f3b44ff85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Jul 16 22:01:01.822399 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:01.822365 2577 patch_prober.go:28] interesting pod/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Jul 16 22:01:01.822580 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:01.822420 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" podUID="bf9dbd76-938f-4670-9d14-1f1f3b44ff85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Jul 16 22:01:02.822528 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:02.822499 2577 patch_prober.go:28] interesting pod/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Jul 16 22:01:02.822916 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:02.822549 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" podUID="bf9dbd76-938f-4670-9d14-1f1f3b44ff85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Jul 16 22:01:03.822165 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:03.822129 2577 patch_prober.go:28] interesting pod/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Jul 16 22:01:03.822374 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:03.822214 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" podUID="bf9dbd76-938f-4670-9d14-1f1f3b44ff85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Jul 16 22:01:04.826078 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:04.826043 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:01:04.827249 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:04.827226 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:01:06.998998 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:06.998967 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9"] Jul 16 22:01:07.002646 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.002632 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.005991 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.005961 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2140ea\"/\"default-dockercfg-c6g54\"" Jul 16 22:01:07.007090 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.007070 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2140ea\"/\"openshift-service-ca.crt\"" Jul 16 22:01:07.007218 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.007119 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2140ea\"/\"kube-root-ca.crt\"" Jul 16 22:01:07.007218 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.007136 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2140ea\"/\"auth-test-model-2140ea-kserve-self-signed-certs\"" Jul 16 22:01:07.018595 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.018573 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9"] Jul 16 22:01:07.089249 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.089220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-dshm\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.089427 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.089273 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-home\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.089427 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.089322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-model-cache\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.089427 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.089404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66273e91-c7ea-418d-89a8-3b92a08268e9-tls-certs\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.089596 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.089433 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mttln\" (UniqueName: \"kubernetes.io/projected/66273e91-c7ea-418d-89a8-3b92a08268e9-kube-api-access-mttln\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.089596 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.089497 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-tmp-dir\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.189872 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.189839 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-home\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.189872 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.189873 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-model-cache\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.190133 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.189911 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66273e91-c7ea-418d-89a8-3b92a08268e9-tls-certs\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.190133 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.189964 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mttln\" (UniqueName: \"kubernetes.io/projected/66273e91-c7ea-418d-89a8-3b92a08268e9-kube-api-access-mttln\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.190133 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.190012 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-tmp-dir\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.190133 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.190046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-dshm\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.190388 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.190336 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-home\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.190388 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.190367 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-tmp-dir\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.190513 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.190495 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-model-cache\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.192269 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.192249 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-dshm\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.192438 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.192421 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66273e91-c7ea-418d-89a8-3b92a08268e9-tls-certs\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.198428 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.198405 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mttln\" (UniqueName: \"kubernetes.io/projected/66273e91-c7ea-418d-89a8-3b92a08268e9-kube-api-access-mttln\") pod \"auth-test-model-2140ea-kserve-78dcb78479-8trc9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.311733 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.311639 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:07.447862 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.447834 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9"] Jul 16 22:01:07.448587 ip-10-0-137-133 kubenswrapper[2577]: W0716 22:01:07.448563 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66273e91_c7ea_418d_89a8_3b92a08268e9.slice/crio-9533c5cac3ecfbf4eb13ab9c3a16da56b53731680e59266ab0d5eda7e68490bf WatchSource:0}: Error finding container 9533c5cac3ecfbf4eb13ab9c3a16da56b53731680e59266ab0d5eda7e68490bf: Status 404 returned error can't find the container with id 9533c5cac3ecfbf4eb13ab9c3a16da56b53731680e59266ab0d5eda7e68490bf Jul 16 22:01:07.848085 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.848050 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" event={"ID":"66273e91-c7ea-418d-89a8-3b92a08268e9","Type":"ContainerStarted","Data":"b9956d6b77eef859916d712a5e2337cc8b1a31f6314fcd291c98099ed79f3fb0"} Jul 16 22:01:07.848085 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.848092 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" event={"ID":"66273e91-c7ea-418d-89a8-3b92a08268e9","Type":"ContainerStarted","Data":"9533c5cac3ecfbf4eb13ab9c3a16da56b53731680e59266ab0d5eda7e68490bf"} Jul 16 22:01:07.868603 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:07.868545 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" podStartSLOduration=1.868529255 podStartE2EDuration="1.868529255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:01:07.865867699 +0000 UTC m=+3084.040975895" watchObservedRunningTime="2026-07-16 22:01:07.868529255 +0000 UTC m=+3084.043637447" Jul 16 22:01:17.311912 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:17.311875 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:17.312384 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:17.311925 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:17.324048 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:17.324025 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:01:17.910835 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:01:17.910805 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:03:32.148644 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:32.148600 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9"] Jul 16 22:03:47.168334 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.168287 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" podUID="66273e91-c7ea-418d-89a8-3b92a08268e9" containerName="main" containerID="cri-o://b9956d6b77eef859916d712a5e2337cc8b1a31f6314fcd291c98099ed79f3fb0" gracePeriod=45 Jul 16 22:03:47.420573 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.420505 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:03:47.553107 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.553069 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-home\") pod \"66273e91-c7ea-418d-89a8-3b92a08268e9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " Jul 16 22:03:47.553107 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.553106 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-model-cache\") pod \"66273e91-c7ea-418d-89a8-3b92a08268e9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " Jul 16 22:03:47.553380 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.553152 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mttln\" (UniqueName: \"kubernetes.io/projected/66273e91-c7ea-418d-89a8-3b92a08268e9-kube-api-access-mttln\") pod \"66273e91-c7ea-418d-89a8-3b92a08268e9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " Jul 16 22:03:47.553380 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.553208 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66273e91-c7ea-418d-89a8-3b92a08268e9-tls-certs\") pod \"66273e91-c7ea-418d-89a8-3b92a08268e9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " Jul 16 22:03:47.553380 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.553237 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-dshm\") pod \"66273e91-c7ea-418d-89a8-3b92a08268e9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " Jul 16 22:03:47.553556 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.553384 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-tmp-dir\") pod \"66273e91-c7ea-418d-89a8-3b92a08268e9\" (UID: \"66273e91-c7ea-418d-89a8-3b92a08268e9\") " Jul 16 22:03:47.553556 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.553387 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-home" (OuterVolumeSpecName: "home") pod "66273e91-c7ea-418d-89a8-3b92a08268e9" (UID: "66273e91-c7ea-418d-89a8-3b92a08268e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:03:47.553556 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.553451 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "66273e91-c7ea-418d-89a8-3b92a08268e9" (UID: "66273e91-c7ea-418d-89a8-3b92a08268e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:03:47.553682 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.553572 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66273e91-c7ea-418d-89a8-3b92a08268e9" (UID: "66273e91-c7ea-418d-89a8-3b92a08268e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:03:47.553825 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.553806 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-home\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:03:47.553890 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.553830 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-model-cache\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:03:47.553890 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.553848 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-tmp-dir\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:03:47.555480 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.555455 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-dshm" (OuterVolumeSpecName: "dshm") pod "66273e91-c7ea-418d-89a8-3b92a08268e9" (UID: "66273e91-c7ea-418d-89a8-3b92a08268e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:03:47.555581 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.555509 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66273e91-c7ea-418d-89a8-3b92a08268e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66273e91-c7ea-418d-89a8-3b92a08268e9" (UID: "66273e91-c7ea-418d-89a8-3b92a08268e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:03:47.555581 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.555550 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66273e91-c7ea-418d-89a8-3b92a08268e9-kube-api-access-mttln" (OuterVolumeSpecName: "kube-api-access-mttln") pod "66273e91-c7ea-418d-89a8-3b92a08268e9" (UID: "66273e91-c7ea-418d-89a8-3b92a08268e9"). InnerVolumeSpecName "kube-api-access-mttln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:03:47.654447 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.654399 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mttln\" (UniqueName: \"kubernetes.io/projected/66273e91-c7ea-418d-89a8-3b92a08268e9-kube-api-access-mttln\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:03:47.654447 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.654439 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66273e91-c7ea-418d-89a8-3b92a08268e9-tls-certs\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:03:47.654447 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.654448 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66273e91-c7ea-418d-89a8-3b92a08268e9-dshm\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:03:47.699045 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.698953 2577 generic.go:356] "Generic (PLEG): container finished" podID="66273e91-c7ea-418d-89a8-3b92a08268e9" containerID="b9956d6b77eef859916d712a5e2337cc8b1a31f6314fcd291c98099ed79f3fb0" exitCode=0 Jul 16 22:03:47.699045 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.699002 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" Jul 16 22:03:47.699045 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.699010 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" event={"ID":"66273e91-c7ea-418d-89a8-3b92a08268e9","Type":"ContainerDied","Data":"b9956d6b77eef859916d712a5e2337cc8b1a31f6314fcd291c98099ed79f3fb0"} Jul 16 22:03:47.699045 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.699041 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9" event={"ID":"66273e91-c7ea-418d-89a8-3b92a08268e9","Type":"ContainerDied","Data":"9533c5cac3ecfbf4eb13ab9c3a16da56b53731680e59266ab0d5eda7e68490bf"} Jul 16 22:03:47.699310 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.699060 2577 scope.go:117] "RemoveContainer" containerID="b9956d6b77eef859916d712a5e2337cc8b1a31f6314fcd291c98099ed79f3fb0" Jul 16 22:03:47.708972 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.708932 2577 scope.go:117] "RemoveContainer" containerID="b9956d6b77eef859916d712a5e2337cc8b1a31f6314fcd291c98099ed79f3fb0" Jul 16 22:03:47.709214 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:03:47.709198 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9956d6b77eef859916d712a5e2337cc8b1a31f6314fcd291c98099ed79f3fb0\": container with ID starting with b9956d6b77eef859916d712a5e2337cc8b1a31f6314fcd291c98099ed79f3fb0 not found: ID does not exist" containerID="b9956d6b77eef859916d712a5e2337cc8b1a31f6314fcd291c98099ed79f3fb0" Jul 16 22:03:47.709274 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.709221 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9956d6b77eef859916d712a5e2337cc8b1a31f6314fcd291c98099ed79f3fb0"} err="failed to get container status \"b9956d6b77eef859916d712a5e2337cc8b1a31f6314fcd291c98099ed79f3fb0\": rpc error: code = NotFound desc = could not find container \"b9956d6b77eef859916d712a5e2337cc8b1a31f6314fcd291c98099ed79f3fb0\": container with ID starting with b9956d6b77eef859916d712a5e2337cc8b1a31f6314fcd291c98099ed79f3fb0 not found: ID does not exist" Jul 16 22:03:47.722042 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.722018 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9"] Jul 16 22:03:47.726871 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:47.726853 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2140ea/auth-test-model-2140ea-kserve-78dcb78479-8trc9"] Jul 16 22:03:48.450969 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:03:48.450929 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66273e91-c7ea-418d-89a8-3b92a08268e9" path="/var/lib/kubelet/pods/66273e91-c7ea-418d-89a8-3b92a08268e9/volumes" Jul 16 22:04:01.799208 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.799153 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf"] Jul 16 22:04:01.799757 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.799552 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66273e91-c7ea-418d-89a8-3b92a08268e9" containerName="main" Jul 16 22:04:01.799757 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.799567 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="66273e91-c7ea-418d-89a8-3b92a08268e9" containerName="main" Jul 16 22:04:01.799757 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.799637 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="66273e91-c7ea-418d-89a8-3b92a08268e9" containerName="main" Jul 16 22:04:01.802875 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.802858 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.807544 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.807521 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a87ce2\"/\"sub-test-model-a87ce2-kserve-self-signed-certs\"" Jul 16 22:04:01.807544 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.807540 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a87ce2\"/\"default-dockercfg-xjdhw\"" Jul 16 22:04:01.807716 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.807522 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a87ce2\"/\"kube-root-ca.crt\"" Jul 16 22:04:01.807716 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.807524 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a87ce2\"/\"openshift-service-ca.crt\"" Jul 16 22:04:01.816428 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.816407 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf"] Jul 16 22:04:01.876362 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.876331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8ps5\" (UniqueName: \"kubernetes.io/projected/b699cc69-3561-472e-aa19-d617ff4761da-kube-api-access-k8ps5\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.876509 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.876369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-tmp-dir\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.876509 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.876393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-dshm\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.876598 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.876535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-model-cache\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.876598 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.876561 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-home\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.876598 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.876589 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b699cc69-3561-472e-aa19-d617ff4761da-tls-certs\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.977824 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.977791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ps5\" (UniqueName: \"kubernetes.io/projected/b699cc69-3561-472e-aa19-d617ff4761da-kube-api-access-k8ps5\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.978025 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.977832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-tmp-dir\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.978025 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.977854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-dshm\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.978025 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.977921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-model-cache\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.978025 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.977940 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-home\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.978025 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.977967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b699cc69-3561-472e-aa19-d617ff4761da-tls-certs\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.978348 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.978320 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-tmp-dir\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.978423 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.978362 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-model-cache\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.978423 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.978411 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-home\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.980199 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.980166 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-dshm\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.980370 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.980354 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b699cc69-3561-472e-aa19-d617ff4761da-tls-certs\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:01.987614 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:01.987591 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ps5\" (UniqueName: \"kubernetes.io/projected/b699cc69-3561-472e-aa19-d617ff4761da-kube-api-access-k8ps5\") pod \"sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:02.112033 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:02.111956 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:02.248044 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:02.248020 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf"] Jul 16 22:04:02.249749 ip-10-0-137-133 kubenswrapper[2577]: W0716 22:04:02.249709 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb699cc69_3561_472e_aa19_d617ff4761da.slice/crio-40b86ac1f565a6daea8fd34280bed9fd4641ced70b94cf727ef3c6584e44fadc WatchSource:0}: Error finding container 40b86ac1f565a6daea8fd34280bed9fd4641ced70b94cf727ef3c6584e44fadc: Status 404 returned error can't find the container with id 40b86ac1f565a6daea8fd34280bed9fd4641ced70b94cf727ef3c6584e44fadc Jul 16 22:04:02.782135 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:02.782099 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" event={"ID":"b699cc69-3561-472e-aa19-d617ff4761da","Type":"ContainerStarted","Data":"930c7d10f1b4f03c7745cf341197bd03a62f7440587b07eebe69bc618db49001"} Jul 16 22:04:02.782135 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:02.782140 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" event={"ID":"b699cc69-3561-472e-aa19-d617ff4761da","Type":"ContainerStarted","Data":"40b86ac1f565a6daea8fd34280bed9fd4641ced70b94cf727ef3c6584e44fadc"} Jul 16 22:04:02.801410 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:02.801360 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" podStartSLOduration=1.801343536 podStartE2EDuration="1.801343536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:04:02.799473453 +0000 UTC m=+3258.974581665" watchObservedRunningTime="2026-07-16 22:04:02.801343536 +0000 UTC m=+3258.976451736" Jul 16 22:04:12.112439 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:12.112394 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:12.112813 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:12.112453 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:12.125131 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:12.125104 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:12.845492 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:12.845462 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:04:44.584073 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:44.584047 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 22:04:44.587362 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:44.587342 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5lc2_1ae55087-4ec5-476b-a36e-a46fc3ae4479/ovn-acl-logging/0.log" Jul 16 22:04:58.595957 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:04:58.595926 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf"] Jul 16 22:05:13.615713 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.615628 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" podUID="b699cc69-3561-472e-aa19-d617ff4761da" containerName="main" containerID="cri-o://930c7d10f1b4f03c7745cf341197bd03a62f7440587b07eebe69bc618db49001" gracePeriod=45 Jul 16 22:05:13.864996 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.864970 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:05:13.924457 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.924428 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b699cc69-3561-472e-aa19-d617ff4761da-tls-certs\") pod \"b699cc69-3561-472e-aa19-d617ff4761da\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " Jul 16 22:05:13.924637 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.924489 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-dshm\") pod \"b699cc69-3561-472e-aa19-d617ff4761da\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " Jul 16 22:05:13.924637 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.924511 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-home\") pod \"b699cc69-3561-472e-aa19-d617ff4761da\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " Jul 16 22:05:13.924637 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.924533 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-tmp-dir\") pod \"b699cc69-3561-472e-aa19-d617ff4761da\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " Jul 16 22:05:13.924637 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.924589 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-model-cache\") pod \"b699cc69-3561-472e-aa19-d617ff4761da\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " Jul 16 22:05:13.924865 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.924669 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8ps5\" (UniqueName: \"kubernetes.io/projected/b699cc69-3561-472e-aa19-d617ff4761da-kube-api-access-k8ps5\") pod \"b699cc69-3561-472e-aa19-d617ff4761da\" (UID: \"b699cc69-3561-472e-aa19-d617ff4761da\") " Jul 16 22:05:13.924865 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.924821 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-home" (OuterVolumeSpecName: "home") pod "b699cc69-3561-472e-aa19-d617ff4761da" (UID: "b699cc69-3561-472e-aa19-d617ff4761da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:05:13.924865 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.924834 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b699cc69-3561-472e-aa19-d617ff4761da" (UID: "b699cc69-3561-472e-aa19-d617ff4761da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:05:13.924865 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.924845 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-model-cache" (OuterVolumeSpecName: "model-cache") pod "b699cc69-3561-472e-aa19-d617ff4761da" (UID: "b699cc69-3561-472e-aa19-d617ff4761da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:05:13.925039 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.925022 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-home\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:05:13.925080 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.925046 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-tmp-dir\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:05:13.925080 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.925059 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-model-cache\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:05:13.926695 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.926673 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b699cc69-3561-472e-aa19-d617ff4761da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b699cc69-3561-472e-aa19-d617ff4761da" (UID: "b699cc69-3561-472e-aa19-d617ff4761da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:05:13.926790 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.926709 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b699cc69-3561-472e-aa19-d617ff4761da-kube-api-access-k8ps5" (OuterVolumeSpecName: "kube-api-access-k8ps5") pod "b699cc69-3561-472e-aa19-d617ff4761da" (UID: "b699cc69-3561-472e-aa19-d617ff4761da"). InnerVolumeSpecName "kube-api-access-k8ps5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:05:13.926790 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:13.926755 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-dshm" (OuterVolumeSpecName: "dshm") pod "b699cc69-3561-472e-aa19-d617ff4761da" (UID: "b699cc69-3561-472e-aa19-d617ff4761da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:05:14.026391 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:14.026354 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b699cc69-3561-472e-aa19-d617ff4761da-dshm\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:05:14.026391 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:14.026385 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8ps5\" (UniqueName: \"kubernetes.io/projected/b699cc69-3561-472e-aa19-d617ff4761da-kube-api-access-k8ps5\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:05:14.026391 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:14.026395 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b699cc69-3561-472e-aa19-d617ff4761da-tls-certs\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:05:14.180786 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:14.180702 2577 generic.go:356] "Generic (PLEG): container finished" podID="b699cc69-3561-472e-aa19-d617ff4761da" containerID="930c7d10f1b4f03c7745cf341197bd03a62f7440587b07eebe69bc618db49001" exitCode=0 Jul 16 22:05:14.180786 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:14.180743 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" Jul 16 22:05:14.180786 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:14.180763 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" event={"ID":"b699cc69-3561-472e-aa19-d617ff4761da","Type":"ContainerDied","Data":"930c7d10f1b4f03c7745cf341197bd03a62f7440587b07eebe69bc618db49001"} Jul 16 22:05:14.181029 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:14.180798 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf" event={"ID":"b699cc69-3561-472e-aa19-d617ff4761da","Type":"ContainerDied","Data":"40b86ac1f565a6daea8fd34280bed9fd4641ced70b94cf727ef3c6584e44fadc"} Jul 16 22:05:14.181029 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:14.180815 2577 scope.go:117] "RemoveContainer" containerID="930c7d10f1b4f03c7745cf341197bd03a62f7440587b07eebe69bc618db49001" Jul 16 22:05:14.190627 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:14.190612 2577 scope.go:117] "RemoveContainer" containerID="930c7d10f1b4f03c7745cf341197bd03a62f7440587b07eebe69bc618db49001" Jul 16 22:05:14.190900 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:05:14.190880 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930c7d10f1b4f03c7745cf341197bd03a62f7440587b07eebe69bc618db49001\": container with ID starting with 930c7d10f1b4f03c7745cf341197bd03a62f7440587b07eebe69bc618db49001 not found: ID does not exist" containerID="930c7d10f1b4f03c7745cf341197bd03a62f7440587b07eebe69bc618db49001" Jul 16 22:05:14.190970 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:14.190911 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930c7d10f1b4f03c7745cf341197bd03a62f7440587b07eebe69bc618db49001"} err="failed to get container status \"930c7d10f1b4f03c7745cf341197bd03a62f7440587b07eebe69bc618db49001\": rpc error: code = NotFound desc = could not find container \"930c7d10f1b4f03c7745cf341197bd03a62f7440587b07eebe69bc618db49001\": container with ID starting with 930c7d10f1b4f03c7745cf341197bd03a62f7440587b07eebe69bc618db49001 not found: ID does not exist" Jul 16 22:05:14.204996 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:14.204964 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf"] Jul 16 22:05:14.210092 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:14.210069 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a87ce2/sub-test-model-a87ce2-kserve-57bc8fb8f9-8z2zf"] Jul 16 22:05:14.452871 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:14.452780 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b699cc69-3561-472e-aa19-d617ff4761da" path="/var/lib/kubelet/pods/b699cc69-3561-472e-aa19-d617ff4761da/volumes" Jul 16 22:05:33.312689 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.312656 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79"] Jul 16 22:05:33.313127 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.313114 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b699cc69-3561-472e-aa19-d617ff4761da" containerName="main" Jul 16 22:05:33.313127 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.313128 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b699cc69-3561-472e-aa19-d617ff4761da" containerName="main" Jul 16 22:05:33.313278 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.313219 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b699cc69-3561-472e-aa19-d617ff4761da" containerName="main" Jul 16 22:05:33.316225 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.316207 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.320448 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.320425 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a87ce2\"/\"kube-root-ca.crt\"" Jul 16 22:05:33.320570 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.320528 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a87ce2\"/\"openshift-service-ca.crt\"" Jul 16 22:05:33.320570 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.320548 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a87ce2\"/\"default-dockercfg-xjdhw\"" Jul 16 22:05:33.320570 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.320556 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a87ce2\"/\"rate-test-model-a87ce2-kserve-self-signed-certs\"" Jul 16 22:05:33.326773 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.326749 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79"] Jul 16 22:05:33.506521 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.506483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-tls-certs\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.506726 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.506562 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95phr\" (UniqueName: \"kubernetes.io/projected/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-kube-api-access-95phr\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.506726 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.506595 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-home\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.506726 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.506620 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-dshm\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.506726 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.506644 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-model-cache\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.506726 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.506682 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-tmp-dir\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.608076 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.607976 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95phr\" (UniqueName: \"kubernetes.io/projected/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-kube-api-access-95phr\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.608076 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.608022 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-home\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.608076 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.608051 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-dshm\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.608076 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.608074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-model-cache\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.608462 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.608117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-tmp-dir\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.608462 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.608226 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-tls-certs\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.608580 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.608538 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-home\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.608643 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.608599 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-model-cache\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.608643 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.608620 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-tmp-dir\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.610460 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.610438 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-dshm\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.610686 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.610668 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-tls-certs\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.617719 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.617691 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95phr\" (UniqueName: \"kubernetes.io/projected/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-kube-api-access-95phr\") pod \"rate-test-model-a87ce2-kserve-856676c947-8xb79\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.625529 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.625509 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:33.761741 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:33.761716 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79"] Jul 16 22:05:33.763070 ip-10-0-137-133 kubenswrapper[2577]: W0716 22:05:33.763040 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5c39baf_4eb3_4d55_ae6e_9516aca4ac24.slice/crio-e62041e50e9bab74f612a375a3dca3435eaca5d356ffa9908f45728621e911e9 WatchSource:0}: Error finding container e62041e50e9bab74f612a375a3dca3435eaca5d356ffa9908f45728621e911e9: Status 404 returned error can't find the container with id e62041e50e9bab74f612a375a3dca3435eaca5d356ffa9908f45728621e911e9 Jul 16 22:05:34.291423 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:34.291383 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" event={"ID":"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24","Type":"ContainerStarted","Data":"88e4cb3586f3141632c52438259eb8eb5e7c384d9f2a7f64b9bfe0be243f1d31"} Jul 16 22:05:34.291605 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:34.291434 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" event={"ID":"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24","Type":"ContainerStarted","Data":"e62041e50e9bab74f612a375a3dca3435eaca5d356ffa9908f45728621e911e9"} Jul 16 22:05:34.313175 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:34.313128 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" podStartSLOduration=1.313111362 podStartE2EDuration="1.313111362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:05:34.311434831 +0000 UTC m=+3350.486543063" watchObservedRunningTime="2026-07-16 22:05:34.313111362 +0000 UTC m=+3350.488219553" Jul 16 22:05:43.626076 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:43.626025 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:43.626627 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:43.626101 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:43.639293 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:43.639265 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:05:44.362301 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:05:44.362275 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:06:21.006421 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:21.006383 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79"] Jul 16 22:06:24.217851 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:24.217812 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t"] Jul 16 22:06:24.221555 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:24.221532 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" Jul 16 22:06:24.224604 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:24.224586 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 22:06:24.230753 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:24.230734 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t"] Jul 16 22:06:24.278815 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:24.278779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t\" (UID: \"b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" Jul 16 22:06:24.380169 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:24.380134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t\" (UID: \"b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" Jul 16 22:06:24.380788 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:24.380769 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t\" (UID: \"b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" Jul 16 22:06:24.532974 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:24.532895 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" Jul 16 22:06:24.661465 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:24.661427 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t"] Jul 16 22:06:24.662973 ip-10-0-137-133 kubenswrapper[2577]: W0716 22:06:24.662940 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58ca41d_b44e_4a9d_ab90_acc77a8cc6bf.slice/crio-1c2e36fe9f625e2b78a25d7d64829f58f2b0c1908ebbfff984da19485b1315fc WatchSource:0}: Error finding container 1c2e36fe9f625e2b78a25d7d64829f58f2b0c1908ebbfff984da19485b1315fc: Status 404 returned error can't find the container with id 1c2e36fe9f625e2b78a25d7d64829f58f2b0c1908ebbfff984da19485b1315fc Jul 16 22:06:25.570118 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:25.570027 2577 generic.go:356] "Generic (PLEG): container finished" podID="b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" containerID="9a348a38bd31b5f89646bc06e64e7fa5132d0002fe29df962b48407397fa685b" exitCode=6 Jul 16 22:06:25.570508 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:25.570111 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" event={"ID":"b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf","Type":"ContainerDied","Data":"9a348a38bd31b5f89646bc06e64e7fa5132d0002fe29df962b48407397fa685b"} Jul 16 22:06:25.570508 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:25.570166 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" event={"ID":"b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf","Type":"ContainerStarted","Data":"1c2e36fe9f625e2b78a25d7d64829f58f2b0c1908ebbfff984da19485b1315fc"} Jul 16 22:06:25.570508 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:25.570345 2577 scope.go:117] "RemoveContainer" containerID="9a348a38bd31b5f89646bc06e64e7fa5132d0002fe29df962b48407397fa685b" Jul 16 22:06:26.577159 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:26.577069 2577 generic.go:356] "Generic (PLEG): container finished" podID="b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" containerID="44d595cf94933cc8cae5e4261188dc8e51f45845246816a75bfdea3d4b0eaa95" exitCode=6 Jul 16 22:06:26.577584 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:26.577152 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" event={"ID":"b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf","Type":"ContainerDied","Data":"44d595cf94933cc8cae5e4261188dc8e51f45845246816a75bfdea3d4b0eaa95"} Jul 16 22:06:26.577584 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:26.577228 2577 scope.go:117] "RemoveContainer" containerID="9a348a38bd31b5f89646bc06e64e7fa5132d0002fe29df962b48407397fa685b" Jul 16 22:06:26.577584 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:26.577492 2577 scope.go:117] "RemoveContainer" containerID="44d595cf94933cc8cae5e4261188dc8e51f45845246816a75bfdea3d4b0eaa95" Jul 16 22:06:26.577724 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:06:26.577707 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t_odh-ai-gateway-infra(b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" podUID="b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" Jul 16 22:06:27.584126 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:27.584098 2577 scope.go:117] "RemoveContainer" containerID="44d595cf94933cc8cae5e4261188dc8e51f45845246816a75bfdea3d4b0eaa95" Jul 16 22:06:27.584516 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:06:27.584315 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t_odh-ai-gateway-infra(b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" podUID="b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" Jul 16 22:06:36.026300 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.026214 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" podUID="e5c39baf-4eb3-4d55-ae6e-9516aca4ac24" containerName="main" containerID="cri-o://88e4cb3586f3141632c52438259eb8eb5e7c384d9f2a7f64b9bfe0be243f1d31" gracePeriod=45 Jul 16 22:06:36.275212 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.275176 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:06:36.333933 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.333858 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x"] Jul 16 22:06:36.334255 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.334242 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c39baf-4eb3-4d55-ae6e-9516aca4ac24" containerName="main" Jul 16 22:06:36.334318 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.334257 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c39baf-4eb3-4d55-ae6e-9516aca4ac24" containerName="main" Jul 16 22:06:36.334355 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.334317 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c39baf-4eb3-4d55-ae6e-9516aca4ac24" containerName="main" Jul 16 22:06:36.339258 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.338251 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" Jul 16 22:06:36.349554 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.349524 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x"] Jul 16 22:06:36.400474 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.400435 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95phr\" (UniqueName: \"kubernetes.io/projected/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-kube-api-access-95phr\") pod \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " Jul 16 22:06:36.400662 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.400532 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-home\") pod \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " Jul 16 22:06:36.400662 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.400555 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-tmp-dir\") pod \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " Jul 16 22:06:36.400662 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.400592 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-tls-certs\") pod \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " Jul 16 22:06:36.400662 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.400632 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-model-cache\") pod \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " Jul 16 22:06:36.400662 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.400657 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-dshm\") pod \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\" (UID: \"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24\") " Jul 16 22:06:36.400926 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.400838 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-home" (OuterVolumeSpecName: "home") pod "e5c39baf-4eb3-4d55-ae6e-9516aca4ac24" (UID: "e5c39baf-4eb3-4d55-ae6e-9516aca4ac24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:06:36.400926 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.400870 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5c39baf-4eb3-4d55-ae6e-9516aca4ac24" (UID: "e5c39baf-4eb3-4d55-ae6e-9516aca4ac24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:06:36.400926 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.400914 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5c39baf-4eb3-4d55-ae6e-9516aca4ac24" (UID: "e5c39baf-4eb3-4d55-ae6e-9516aca4ac24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:06:36.401083 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.400938 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e4f4cd-9a33-461d-b345-6f08b1e8244a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x\" (UID: \"e3e4f4cd-9a33-461d-b345-6f08b1e8244a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" Jul 16 22:06:36.401083 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.401011 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-home\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:06:36.401083 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.401026 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-tmp-dir\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:06:36.401083 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.401039 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-model-cache\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:06:36.402661 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.402630 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5c39baf-4eb3-4d55-ae6e-9516aca4ac24" (UID: "e5c39baf-4eb3-4d55-ae6e-9516aca4ac24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:06:36.402780 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.402670 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-dshm" (OuterVolumeSpecName: "dshm") pod "e5c39baf-4eb3-4d55-ae6e-9516aca4ac24" (UID: "e5c39baf-4eb3-4d55-ae6e-9516aca4ac24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:06:36.402780 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.402696 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-kube-api-access-95phr" (OuterVolumeSpecName: "kube-api-access-95phr") pod "e5c39baf-4eb3-4d55-ae6e-9516aca4ac24" (UID: "e5c39baf-4eb3-4d55-ae6e-9516aca4ac24"). InnerVolumeSpecName "kube-api-access-95phr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:06:36.501780 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.501743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e4f4cd-9a33-461d-b345-6f08b1e8244a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x\" (UID: \"e3e4f4cd-9a33-461d-b345-6f08b1e8244a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" Jul 16 22:06:36.501945 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.501815 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-dshm\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:06:36.501945 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.501832 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95phr\" (UniqueName: \"kubernetes.io/projected/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-kube-api-access-95phr\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:06:36.501945 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.501845 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24-tls-certs\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:06:36.502393 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.502376 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e4f4cd-9a33-461d-b345-6f08b1e8244a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x\" (UID: \"e3e4f4cd-9a33-461d-b345-6f08b1e8244a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" Jul 16 22:06:36.642969 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.642929 2577 generic.go:356] "Generic (PLEG): container finished" podID="e5c39baf-4eb3-4d55-ae6e-9516aca4ac24" containerID="88e4cb3586f3141632c52438259eb8eb5e7c384d9f2a7f64b9bfe0be243f1d31" exitCode=0 Jul 16 22:06:36.643130 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.643009 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" Jul 16 22:06:36.643130 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.643018 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" event={"ID":"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24","Type":"ContainerDied","Data":"88e4cb3586f3141632c52438259eb8eb5e7c384d9f2a7f64b9bfe0be243f1d31"} Jul 16 22:06:36.643130 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.643065 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79" event={"ID":"e5c39baf-4eb3-4d55-ae6e-9516aca4ac24","Type":"ContainerDied","Data":"e62041e50e9bab74f612a375a3dca3435eaca5d356ffa9908f45728621e911e9"} Jul 16 22:06:36.643130 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.643085 2577 scope.go:117] "RemoveContainer" containerID="88e4cb3586f3141632c52438259eb8eb5e7c384d9f2a7f64b9bfe0be243f1d31" Jul 16 22:06:36.651564 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.651546 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" Jul 16 22:06:36.652389 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.652372 2577 scope.go:117] "RemoveContainer" containerID="88e4cb3586f3141632c52438259eb8eb5e7c384d9f2a7f64b9bfe0be243f1d31" Jul 16 22:06:36.652659 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:06:36.652638 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e4cb3586f3141632c52438259eb8eb5e7c384d9f2a7f64b9bfe0be243f1d31\": container with ID starting with 88e4cb3586f3141632c52438259eb8eb5e7c384d9f2a7f64b9bfe0be243f1d31 not found: ID does not exist" containerID="88e4cb3586f3141632c52438259eb8eb5e7c384d9f2a7f64b9bfe0be243f1d31" Jul 16 22:06:36.652763 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.652664 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e4cb3586f3141632c52438259eb8eb5e7c384d9f2a7f64b9bfe0be243f1d31"} err="failed to get container status \"88e4cb3586f3141632c52438259eb8eb5e7c384d9f2a7f64b9bfe0be243f1d31\": rpc error: code = NotFound desc = could not find container \"88e4cb3586f3141632c52438259eb8eb5e7c384d9f2a7f64b9bfe0be243f1d31\": container with ID starting with 88e4cb3586f3141632c52438259eb8eb5e7c384d9f2a7f64b9bfe0be243f1d31 not found: ID does not exist" Jul 16 22:06:36.663282 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.663251 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79"] Jul 16 22:06:36.666688 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.666669 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a87ce2/rate-test-model-a87ce2-kserve-856676c947-8xb79"] Jul 16 22:06:36.783402 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:36.783376 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x"] Jul 16 22:06:36.784126 ip-10-0-137-133 kubenswrapper[2577]: W0716 22:06:36.784106 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e4f4cd_9a33_461d_b345_6f08b1e8244a.slice/crio-049ff46bc7eb7dfdce3b0eba1c7b25fb931e1e9cf3e60f9b66cdd299519888cd WatchSource:0}: Error finding container 049ff46bc7eb7dfdce3b0eba1c7b25fb931e1e9cf3e60f9b66cdd299519888cd: Status 404 returned error can't find the container with id 049ff46bc7eb7dfdce3b0eba1c7b25fb931e1e9cf3e60f9b66cdd299519888cd Jul 16 22:06:37.651019 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:37.650912 2577 generic.go:356] "Generic (PLEG): container finished" podID="e3e4f4cd-9a33-461d-b345-6f08b1e8244a" containerID="5f66db835b379eb06c87a2b2b8e2a55d6d85751def6fd59106e99286768108b5" exitCode=6 Jul 16 22:06:37.651019 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:37.650978 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" event={"ID":"e3e4f4cd-9a33-461d-b345-6f08b1e8244a","Type":"ContainerDied","Data":"5f66db835b379eb06c87a2b2b8e2a55d6d85751def6fd59106e99286768108b5"} Jul 16 22:06:37.651481 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:37.651036 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" event={"ID":"e3e4f4cd-9a33-461d-b345-6f08b1e8244a","Type":"ContainerStarted","Data":"049ff46bc7eb7dfdce3b0eba1c7b25fb931e1e9cf3e60f9b66cdd299519888cd"} Jul 16 22:06:37.651481 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:37.651231 2577 scope.go:117] "RemoveContainer" containerID="5f66db835b379eb06c87a2b2b8e2a55d6d85751def6fd59106e99286768108b5" Jul 16 22:06:38.451645 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:38.451603 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5c39baf-4eb3-4d55-ae6e-9516aca4ac24" path="/var/lib/kubelet/pods/e5c39baf-4eb3-4d55-ae6e-9516aca4ac24/volumes" Jul 16 22:06:38.658506 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:38.658472 2577 generic.go:356] "Generic (PLEG): container finished" podID="e3e4f4cd-9a33-461d-b345-6f08b1e8244a" containerID="c6ec7204d1b4167398ebb7779bd67794f0acafe2725ab8775d79d4ce73169957" exitCode=6 Jul 16 22:06:38.658991 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:38.658539 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" event={"ID":"e3e4f4cd-9a33-461d-b345-6f08b1e8244a","Type":"ContainerDied","Data":"c6ec7204d1b4167398ebb7779bd67794f0acafe2725ab8775d79d4ce73169957"} Jul 16 22:06:38.658991 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:38.658598 2577 scope.go:117] "RemoveContainer" containerID="5f66db835b379eb06c87a2b2b8e2a55d6d85751def6fd59106e99286768108b5" Jul 16 22:06:38.658991 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:38.658836 2577 scope.go:117] "RemoveContainer" containerID="c6ec7204d1b4167398ebb7779bd67794f0acafe2725ab8775d79d4ce73169957" Jul 16 22:06:38.659162 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:06:38.659093 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x_odh-ai-gateway-infra(e3e4f4cd-9a33-461d-b345-6f08b1e8244a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" podUID="e3e4f4cd-9a33-461d-b345-6f08b1e8244a" Jul 16 22:06:39.665742 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:39.665713 2577 scope.go:117] "RemoveContainer" containerID="c6ec7204d1b4167398ebb7779bd67794f0acafe2725ab8775d79d4ce73169957" Jul 16 22:06:39.666230 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:06:39.665953 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x_odh-ai-gateway-infra(e3e4f4cd-9a33-461d-b345-6f08b1e8244a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" podUID="e3e4f4cd-9a33-461d-b345-6f08b1e8244a" Jul 16 22:06:41.445257 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:41.445225 2577 scope.go:117] "RemoveContainer" containerID="44d595cf94933cc8cae5e4261188dc8e51f45845246816a75bfdea3d4b0eaa95" Jul 16 22:06:41.680048 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:41.680011 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" event={"ID":"b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf","Type":"ContainerStarted","Data":"f2bc77307c6005e4b69e70754c20285d7e16c63fe3834ef39b93d2921205805f"} Jul 16 22:06:41.698306 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:41.698210 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" podStartSLOduration=17.698172586 podStartE2EDuration="17.698172586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:06:41.697636961 +0000 UTC m=+3417.872745174" watchObservedRunningTime="2026-07-16 22:06:41.698172586 +0000 UTC m=+3417.873280779" Jul 16 22:06:42.688520 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:42.688489 2577 generic.go:356] "Generic (PLEG): container finished" podID="b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" containerID="f2bc77307c6005e4b69e70754c20285d7e16c63fe3834ef39b93d2921205805f" exitCode=6 Jul 16 22:06:42.688950 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:42.688540 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" event={"ID":"b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf","Type":"ContainerDied","Data":"f2bc77307c6005e4b69e70754c20285d7e16c63fe3834ef39b93d2921205805f"} Jul 16 22:06:42.688950 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:42.688589 2577 scope.go:117] "RemoveContainer" containerID="44d595cf94933cc8cae5e4261188dc8e51f45845246816a75bfdea3d4b0eaa95" Jul 16 22:06:42.688950 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:42.688881 2577 scope.go:117] "RemoveContainer" containerID="f2bc77307c6005e4b69e70754c20285d7e16c63fe3834ef39b93d2921205805f" Jul 16 22:06:42.689144 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:06:42.689124 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t_odh-ai-gateway-infra(b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" podUID="b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" Jul 16 22:06:42.709644 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:42.709615 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t"] Jul 16 22:06:43.830624 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:43.830598 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" Jul 16 22:06:43.969294 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:43.969203 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf-maas-api-service-ca\") pod \"b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf\" (UID: \"b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf\") " Jul 16 22:06:43.969522 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:43.969500 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" (UID: "b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:06:44.070645 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:44.070608 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:06:44.701612 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:44.701585 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" Jul 16 22:06:44.701782 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:44.701578 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t" event={"ID":"b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf","Type":"ContainerDied","Data":"1c2e36fe9f625e2b78a25d7d64829f58f2b0c1908ebbfff984da19485b1315fc"} Jul 16 22:06:44.701782 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:44.701719 2577 scope.go:117] "RemoveContainer" containerID="f2bc77307c6005e4b69e70754c20285d7e16c63fe3834ef39b93d2921205805f" Jul 16 22:06:44.728920 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:44.728895 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t"] Jul 16 22:06:44.732374 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:44.732354 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1cf683e-b2q8t"] Jul 16 22:06:46.451161 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:46.451125 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" path="/var/lib/kubelet/pods/b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf/volumes" Jul 16 22:06:51.030566 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.030537 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj"] Jul 16 22:06:51.031015 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.030738 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" podUID="ae1561f4-851b-46a1-bd8c-1778d5ef8b0f" containerName="maas-api" containerID="cri-o://79e43c2e04d6db0983c27810524ffad217681f39416b11a2ef0d04342e0acadb" gracePeriod=30 Jul 16 22:06:51.282041 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.281983 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" Jul 16 22:06:51.443345 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.443314 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-openshift-service-ca\") pod \"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f\" (UID: \"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f\") " Jul 16 22:06:51.443345 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.443345 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-maas-api-tls\") pod \"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f\" (UID: \"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f\") " Jul 16 22:06:51.443611 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.443369 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77kmd\" (UniqueName: \"kubernetes.io/projected/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-kube-api-access-77kmd\") pod \"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f\" (UID: \"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f\") " Jul 16 22:06:51.443766 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.443743 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ae1561f4-851b-46a1-bd8c-1778d5ef8b0f" (UID: "ae1561f4-851b-46a1-bd8c-1778d5ef8b0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:06:51.445161 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.445144 2577 scope.go:117] "RemoveContainer" containerID="c6ec7204d1b4167398ebb7779bd67794f0acafe2725ab8775d79d4ce73169957" Jul 16 22:06:51.445680 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.445656 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ae1561f4-851b-46a1-bd8c-1778d5ef8b0f" (UID: "ae1561f4-851b-46a1-bd8c-1778d5ef8b0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:06:51.445751 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.445683 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-kube-api-access-77kmd" (OuterVolumeSpecName: "kube-api-access-77kmd") pod "ae1561f4-851b-46a1-bd8c-1778d5ef8b0f" (UID: "ae1561f4-851b-46a1-bd8c-1778d5ef8b0f"). InnerVolumeSpecName "kube-api-access-77kmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:06:51.545032 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.544947 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-openshift-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:06:51.545032 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.544976 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-maas-api-tls\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:06:51.545032 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.544985 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77kmd\" (UniqueName: \"kubernetes.io/projected/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f-kube-api-access-77kmd\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:06:51.739417 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.739387 2577 generic.go:356] "Generic (PLEG): container finished" podID="ae1561f4-851b-46a1-bd8c-1778d5ef8b0f" containerID="79e43c2e04d6db0983c27810524ffad217681f39416b11a2ef0d04342e0acadb" exitCode=0 Jul 16 22:06:51.739538 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.739429 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" Jul 16 22:06:51.739538 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.739469 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" event={"ID":"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f","Type":"ContainerDied","Data":"79e43c2e04d6db0983c27810524ffad217681f39416b11a2ef0d04342e0acadb"} Jul 16 22:06:51.739538 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.739527 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj" event={"ID":"ae1561f4-851b-46a1-bd8c-1778d5ef8b0f","Type":"ContainerDied","Data":"33ca6b0d758936faffbcad5ae1d858321bb7ff0b6bf3774e3abc5762d46af90e"} Jul 16 22:06:51.739702 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.739562 2577 scope.go:117] "RemoveContainer" containerID="79e43c2e04d6db0983c27810524ffad217681f39416b11a2ef0d04342e0acadb" Jul 16 22:06:51.741598 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.741568 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" event={"ID":"e3e4f4cd-9a33-461d-b345-6f08b1e8244a","Type":"ContainerStarted","Data":"7dce4a23033d9e400347f13c89557841ce3fd4088677b420befdce608bdbbfc7"} Jul 16 22:06:51.748209 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.748170 2577 scope.go:117] "RemoveContainer" containerID="79e43c2e04d6db0983c27810524ffad217681f39416b11a2ef0d04342e0acadb" Jul 16 22:06:51.748426 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:06:51.748410 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e43c2e04d6db0983c27810524ffad217681f39416b11a2ef0d04342e0acadb\": container with ID starting with 79e43c2e04d6db0983c27810524ffad217681f39416b11a2ef0d04342e0acadb not found: ID does not exist" containerID="79e43c2e04d6db0983c27810524ffad217681f39416b11a2ef0d04342e0acadb" Jul 16 22:06:51.748475 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.748433 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e43c2e04d6db0983c27810524ffad217681f39416b11a2ef0d04342e0acadb"} err="failed to get container status \"79e43c2e04d6db0983c27810524ffad217681f39416b11a2ef0d04342e0acadb\": rpc error: code = NotFound desc = could not find container \"79e43c2e04d6db0983c27810524ffad217681f39416b11a2ef0d04342e0acadb\": container with ID starting with 79e43c2e04d6db0983c27810524ffad217681f39416b11a2ef0d04342e0acadb not found: ID does not exist" Jul 16 22:06:51.780702 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.780676 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj"] Jul 16 22:06:51.794374 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.794350 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2140ea-6f978957dc-t9hmj"] Jul 16 22:06:51.827622 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:51.827523 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" podStartSLOduration=15.827509741 podStartE2EDuration="15.827509741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:06:51.825548988 +0000 UTC m=+3428.000657205" watchObservedRunningTime="2026-07-16 22:06:51.827509741 +0000 UTC m=+3428.002617933" Jul 16 22:06:52.450909 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:52.450874 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae1561f4-851b-46a1-bd8c-1778d5ef8b0f" path="/var/lib/kubelet/pods/ae1561f4-851b-46a1-bd8c-1778d5ef8b0f/volumes" Jul 16 22:06:52.749282 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:52.749179 2577 generic.go:356] "Generic (PLEG): container finished" podID="e3e4f4cd-9a33-461d-b345-6f08b1e8244a" containerID="7dce4a23033d9e400347f13c89557841ce3fd4088677b420befdce608bdbbfc7" exitCode=6 Jul 16 22:06:52.749282 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:52.749250 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" event={"ID":"e3e4f4cd-9a33-461d-b345-6f08b1e8244a","Type":"ContainerDied","Data":"7dce4a23033d9e400347f13c89557841ce3fd4088677b420befdce608bdbbfc7"} Jul 16 22:06:52.749484 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:52.749308 2577 scope.go:117] "RemoveContainer" containerID="c6ec7204d1b4167398ebb7779bd67794f0acafe2725ab8775d79d4ce73169957" Jul 16 22:06:52.749610 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:52.749583 2577 scope.go:117] "RemoveContainer" containerID="7dce4a23033d9e400347f13c89557841ce3fd4088677b420befdce608bdbbfc7" Jul 16 22:06:52.749836 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:06:52.749811 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x_odh-ai-gateway-infra(e3e4f4cd-9a33-461d-b345-6f08b1e8244a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" podUID="e3e4f4cd-9a33-461d-b345-6f08b1e8244a" Jul 16 22:06:52.831457 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:52.831423 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x"] Jul 16 22:06:53.897388 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:53.897359 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" Jul 16 22:06:54.068213 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:54.068096 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e4f4cd-9a33-461d-b345-6f08b1e8244a-maas-api-service-ca\") pod \"e3e4f4cd-9a33-461d-b345-6f08b1e8244a\" (UID: \"e3e4f4cd-9a33-461d-b345-6f08b1e8244a\") " Jul 16 22:06:54.068534 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:54.068505 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e4f4cd-9a33-461d-b345-6f08b1e8244a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e3e4f4cd-9a33-461d-b345-6f08b1e8244a" (UID: "e3e4f4cd-9a33-461d-b345-6f08b1e8244a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:06:54.169271 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:54.169231 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e4f4cd-9a33-461d-b345-6f08b1e8244a-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:06:54.762420 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:54.762331 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" Jul 16 22:06:54.762420 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:54.762364 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x" event={"ID":"e3e4f4cd-9a33-461d-b345-6f08b1e8244a","Type":"ContainerDied","Data":"049ff46bc7eb7dfdce3b0eba1c7b25fb931e1e9cf3e60f9b66cdd299519888cd"} Jul 16 22:06:54.762666 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:54.762441 2577 scope.go:117] "RemoveContainer" containerID="7dce4a23033d9e400347f13c89557841ce3fd4088677b420befdce608bdbbfc7" Jul 16 22:06:54.783687 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:54.783663 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x"] Jul 16 22:06:54.787526 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:54.787504 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-8pd9x"] Jul 16 22:06:56.450353 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.450313 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e4f4cd-9a33-461d-b345-6f08b1e8244a" path="/var/lib/kubelet/pods/e3e4f4cd-9a33-461d-b345-6f08b1e8244a/volumes" Jul 16 22:06:56.670010 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.669970 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft"] Jul 16 22:06:56.670421 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670404 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" containerName="revoke-keys" Jul 16 22:06:56.670507 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670424 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" containerName="revoke-keys" Jul 16 22:06:56.670507 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670436 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" containerName="revoke-keys" Jul 16 22:06:56.670507 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670445 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" containerName="revoke-keys" Jul 16 22:06:56.670507 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670460 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e4f4cd-9a33-461d-b345-6f08b1e8244a" containerName="revoke-keys" Jul 16 22:06:56.670507 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670470 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e4f4cd-9a33-461d-b345-6f08b1e8244a" containerName="revoke-keys" Jul 16 22:06:56.670507 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670498 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae1561f4-851b-46a1-bd8c-1778d5ef8b0f" containerName="maas-api" Jul 16 22:06:56.670507 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670506 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae1561f4-851b-46a1-bd8c-1778d5ef8b0f" containerName="maas-api" Jul 16 22:06:56.670863 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670519 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e4f4cd-9a33-461d-b345-6f08b1e8244a" containerName="revoke-keys" Jul 16 22:06:56.670863 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670529 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e4f4cd-9a33-461d-b345-6f08b1e8244a" containerName="revoke-keys" Jul 16 22:06:56.670863 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670544 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e4f4cd-9a33-461d-b345-6f08b1e8244a" containerName="revoke-keys" Jul 16 22:06:56.670863 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670553 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e4f4cd-9a33-461d-b345-6f08b1e8244a" containerName="revoke-keys" Jul 16 22:06:56.670863 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670659 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae1561f4-851b-46a1-bd8c-1778d5ef8b0f" containerName="maas-api" Jul 16 22:06:56.670863 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670674 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" containerName="revoke-keys" Jul 16 22:06:56.670863 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670683 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e4f4cd-9a33-461d-b345-6f08b1e8244a" containerName="revoke-keys" Jul 16 22:06:56.670863 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670692 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e4f4cd-9a33-461d-b345-6f08b1e8244a" containerName="revoke-keys" Jul 16 22:06:56.670863 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670701 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" containerName="revoke-keys" Jul 16 22:06:56.670863 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.670708 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e4f4cd-9a33-461d-b345-6f08b1e8244a" containerName="revoke-keys" Jul 16 22:06:56.675384 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.675363 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" Jul 16 22:06:56.678536 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.678514 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 22:06:56.681606 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.681584 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft"] Jul 16 22:06:56.793056 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.792964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b53cd43-a6ad-4b93-a065-3aaf73e4935e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft\" (UID: \"1b53cd43-a6ad-4b93-a065-3aaf73e4935e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" Jul 16 22:06:56.894536 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.894487 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b53cd43-a6ad-4b93-a065-3aaf73e4935e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft\" (UID: \"1b53cd43-a6ad-4b93-a065-3aaf73e4935e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" Jul 16 22:06:56.895080 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.895060 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b53cd43-a6ad-4b93-a065-3aaf73e4935e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft\" (UID: \"1b53cd43-a6ad-4b93-a065-3aaf73e4935e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" Jul 16 22:06:56.987928 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:56.987891 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" Jul 16 22:06:57.144448 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:57.144418 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft"] Jul 16 22:06:57.145810 ip-10-0-137-133 kubenswrapper[2577]: W0716 22:06:57.145781 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b53cd43_a6ad_4b93_a065_3aaf73e4935e.slice/crio-8cbec4bbac7da73e120c528d70318ef295fb96e0fe15b371dda47de8848841a7 WatchSource:0}: Error finding container 8cbec4bbac7da73e120c528d70318ef295fb96e0fe15b371dda47de8848841a7: Status 404 returned error can't find the container with id 8cbec4bbac7da73e120c528d70318ef295fb96e0fe15b371dda47de8848841a7 Jul 16 22:06:57.780081 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:57.779988 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" event={"ID":"1b53cd43-a6ad-4b93-a065-3aaf73e4935e","Type":"ContainerStarted","Data":"a7b9c40f7d502231414f4b8e0aad63e1ea020553674cd6ebc9f53b54b80955d9"} Jul 16 22:06:57.780081 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:57.780052 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" event={"ID":"1b53cd43-a6ad-4b93-a065-3aaf73e4935e","Type":"ContainerStarted","Data":"8cbec4bbac7da73e120c528d70318ef295fb96e0fe15b371dda47de8848841a7"} Jul 16 22:06:57.798113 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:57.798063 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" podStartSLOduration=1.798048603 podStartE2EDuration="1.798048603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:06:57.796779007 +0000 UTC m=+3433.971887200" watchObservedRunningTime="2026-07-16 22:06:57.798048603 +0000 UTC m=+3433.973156795" Jul 16 22:06:57.910409 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:57.910374 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl"] Jul 16 22:06:57.910820 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:57.910802 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" containerName="revoke-keys" Jul 16 22:06:57.910912 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:57.910821 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" containerName="revoke-keys" Jul 16 22:06:57.910912 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:57.910906 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b58ca41d-b44e-4a9d-ab90-acc77a8cc6bf" containerName="revoke-keys" Jul 16 22:06:57.913840 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:57.913822 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" Jul 16 22:06:57.921397 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:57.921375 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl"] Jul 16 22:06:58.003118 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:58.003085 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba94b701-3810-4a34-b5c2-b1979dd8cbc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl\" (UID: \"ba94b701-3810-4a34-b5c2-b1979dd8cbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" Jul 16 22:06:58.103625 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:58.103541 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba94b701-3810-4a34-b5c2-b1979dd8cbc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl\" (UID: \"ba94b701-3810-4a34-b5c2-b1979dd8cbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" Jul 16 22:06:58.104133 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:58.104115 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba94b701-3810-4a34-b5c2-b1979dd8cbc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl\" (UID: \"ba94b701-3810-4a34-b5c2-b1979dd8cbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" Jul 16 22:06:58.256704 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:58.256668 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" Jul 16 22:06:58.392441 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:58.392417 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl"] Jul 16 22:06:58.393907 ip-10-0-137-133 kubenswrapper[2577]: W0716 22:06:58.393878 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba94b701_3810_4a34_b5c2_b1979dd8cbc1.slice/crio-5f5c659acdc3c7d1f6b94a6a734e6489b705cc395e360c251825b1878b588572 WatchSource:0}: Error finding container 5f5c659acdc3c7d1f6b94a6a734e6489b705cc395e360c251825b1878b588572: Status 404 returned error can't find the container with id 5f5c659acdc3c7d1f6b94a6a734e6489b705cc395e360c251825b1878b588572 Jul 16 22:06:58.786626 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:58.786592 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" event={"ID":"ba94b701-3810-4a34-b5c2-b1979dd8cbc1","Type":"ContainerStarted","Data":"3c446c96798c0f8dbf55266ca91a0b5c7e682bc34288b9127dd49bba1608f0eb"} Jul 16 22:06:58.787081 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:58.786652 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" event={"ID":"ba94b701-3810-4a34-b5c2-b1979dd8cbc1","Type":"ContainerStarted","Data":"5f5c659acdc3c7d1f6b94a6a734e6489b705cc395e360c251825b1878b588572"} Jul 16 22:06:58.788068 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:58.788044 2577 generic.go:356] "Generic (PLEG): container finished" podID="1b53cd43-a6ad-4b93-a065-3aaf73e4935e" containerID="a7b9c40f7d502231414f4b8e0aad63e1ea020553674cd6ebc9f53b54b80955d9" exitCode=6 Jul 16 22:06:58.788205 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:58.788078 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" event={"ID":"1b53cd43-a6ad-4b93-a065-3aaf73e4935e","Type":"ContainerDied","Data":"a7b9c40f7d502231414f4b8e0aad63e1ea020553674cd6ebc9f53b54b80955d9"} Jul 16 22:06:58.788336 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:58.788320 2577 scope.go:117] "RemoveContainer" containerID="a7b9c40f7d502231414f4b8e0aad63e1ea020553674cd6ebc9f53b54b80955d9" Jul 16 22:06:58.804548 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:58.804508 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" podStartSLOduration=1.8044890850000002 podStartE2EDuration="1.804489085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:06:58.803657665 +0000 UTC m=+3434.978765859" watchObservedRunningTime="2026-07-16 22:06:58.804489085 +0000 UTC m=+3434.979597278" Jul 16 22:06:59.795092 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:59.795051 2577 generic.go:356] "Generic (PLEG): container finished" podID="1b53cd43-a6ad-4b93-a065-3aaf73e4935e" containerID="c37cc531a3d87486f7bb4e2afc9e7d25d748ec335fdd2f757e91e42bc9f1c269" exitCode=6 Jul 16 22:06:59.795527 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:59.795128 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" event={"ID":"1b53cd43-a6ad-4b93-a065-3aaf73e4935e","Type":"ContainerDied","Data":"c37cc531a3d87486f7bb4e2afc9e7d25d748ec335fdd2f757e91e42bc9f1c269"} Jul 16 22:06:59.795527 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:59.795207 2577 scope.go:117] "RemoveContainer" containerID="a7b9c40f7d502231414f4b8e0aad63e1ea020553674cd6ebc9f53b54b80955d9" Jul 16 22:06:59.795527 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:59.795378 2577 scope.go:117] "RemoveContainer" containerID="c37cc531a3d87486f7bb4e2afc9e7d25d748ec335fdd2f757e91e42bc9f1c269" Jul 16 22:06:59.795765 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:06:59.795618 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft_odh-ai-gateway-infra(1b53cd43-a6ad-4b93-a065-3aaf73e4935e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" podUID="1b53cd43-a6ad-4b93-a065-3aaf73e4935e" Jul 16 22:06:59.796862 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:59.796841 2577 generic.go:356] "Generic (PLEG): container finished" podID="ba94b701-3810-4a34-b5c2-b1979dd8cbc1" containerID="3c446c96798c0f8dbf55266ca91a0b5c7e682bc34288b9127dd49bba1608f0eb" exitCode=6 Jul 16 22:06:59.796965 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:59.796922 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" event={"ID":"ba94b701-3810-4a34-b5c2-b1979dd8cbc1","Type":"ContainerDied","Data":"3c446c96798c0f8dbf55266ca91a0b5c7e682bc34288b9127dd49bba1608f0eb"} Jul 16 22:06:59.797085 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:06:59.797073 2577 scope.go:117] "RemoveContainer" containerID="3c446c96798c0f8dbf55266ca91a0b5c7e682bc34288b9127dd49bba1608f0eb" Jul 16 22:07:00.804080 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:00.803995 2577 generic.go:356] "Generic (PLEG): container finished" podID="ba94b701-3810-4a34-b5c2-b1979dd8cbc1" containerID="f0787cf889d747c5f94be1fc8443b5dce0d245d161efca2f2d5e98cdba1bad82" exitCode=6 Jul 16 22:07:00.804537 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:00.804093 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" event={"ID":"ba94b701-3810-4a34-b5c2-b1979dd8cbc1","Type":"ContainerDied","Data":"f0787cf889d747c5f94be1fc8443b5dce0d245d161efca2f2d5e98cdba1bad82"} Jul 16 22:07:00.804537 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:00.804157 2577 scope.go:117] "RemoveContainer" containerID="3c446c96798c0f8dbf55266ca91a0b5c7e682bc34288b9127dd49bba1608f0eb" Jul 16 22:07:00.804537 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:00.804304 2577 scope.go:117] "RemoveContainer" containerID="f0787cf889d747c5f94be1fc8443b5dce0d245d161efca2f2d5e98cdba1bad82" Jul 16 22:07:00.804537 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:07:00.804530 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl_odh-ai-gateway-infra(ba94b701-3810-4a34-b5c2-b1979dd8cbc1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" podUID="ba94b701-3810-4a34-b5c2-b1979dd8cbc1" Jul 16 22:07:00.806002 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:00.805989 2577 scope.go:117] "RemoveContainer" containerID="c37cc531a3d87486f7bb4e2afc9e7d25d748ec335fdd2f757e91e42bc9f1c269" Jul 16 22:07:00.806225 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:07:00.806134 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft_odh-ai-gateway-infra(1b53cd43-a6ad-4b93-a065-3aaf73e4935e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" podUID="1b53cd43-a6ad-4b93-a065-3aaf73e4935e" Jul 16 22:07:01.813061 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:01.813028 2577 scope.go:117] "RemoveContainer" containerID="f0787cf889d747c5f94be1fc8443b5dce0d245d161efca2f2d5e98cdba1bad82" Jul 16 22:07:01.813482 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:07:01.813251 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl_odh-ai-gateway-infra(ba94b701-3810-4a34-b5c2-b1979dd8cbc1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" podUID="ba94b701-3810-4a34-b5c2-b1979dd8cbc1" Jul 16 22:07:12.445599 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:12.445565 2577 scope.go:117] "RemoveContainer" containerID="f0787cf889d747c5f94be1fc8443b5dce0d245d161efca2f2d5e98cdba1bad82" Jul 16 22:07:12.446073 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:12.445648 2577 scope.go:117] "RemoveContainer" containerID="c37cc531a3d87486f7bb4e2afc9e7d25d748ec335fdd2f757e91e42bc9f1c269" Jul 16 22:07:12.873778 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:12.873746 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" event={"ID":"1b53cd43-a6ad-4b93-a065-3aaf73e4935e","Type":"ContainerStarted","Data":"0581478a4ce20b5b4004995ede877b1daa435da79c726f6062ca9835124b68ed"} Jul 16 22:07:12.875578 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:12.875550 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" event={"ID":"ba94b701-3810-4a34-b5c2-b1979dd8cbc1","Type":"ContainerStarted","Data":"21afa91cf9bb6117a64e51acb7387148118c6b2389c43b53b98420be7c29343d"} Jul 16 22:07:13.882593 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:13.882559 2577 generic.go:356] "Generic (PLEG): container finished" podID="1b53cd43-a6ad-4b93-a065-3aaf73e4935e" containerID="0581478a4ce20b5b4004995ede877b1daa435da79c726f6062ca9835124b68ed" exitCode=6 Jul 16 22:07:13.883039 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:13.882635 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" event={"ID":"1b53cd43-a6ad-4b93-a065-3aaf73e4935e","Type":"ContainerDied","Data":"0581478a4ce20b5b4004995ede877b1daa435da79c726f6062ca9835124b68ed"} Jul 16 22:07:13.883039 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:13.882696 2577 scope.go:117] "RemoveContainer" containerID="c37cc531a3d87486f7bb4e2afc9e7d25d748ec335fdd2f757e91e42bc9f1c269" Jul 16 22:07:13.883039 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:13.882977 2577 scope.go:117] "RemoveContainer" containerID="0581478a4ce20b5b4004995ede877b1daa435da79c726f6062ca9835124b68ed" Jul 16 22:07:13.883260 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:07:13.883227 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft_odh-ai-gateway-infra(1b53cd43-a6ad-4b93-a065-3aaf73e4935e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" podUID="1b53cd43-a6ad-4b93-a065-3aaf73e4935e" Jul 16 22:07:13.884472 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:13.884452 2577 generic.go:356] "Generic (PLEG): container finished" podID="ba94b701-3810-4a34-b5c2-b1979dd8cbc1" containerID="21afa91cf9bb6117a64e51acb7387148118c6b2389c43b53b98420be7c29343d" exitCode=6 Jul 16 22:07:13.884572 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:13.884522 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" event={"ID":"ba94b701-3810-4a34-b5c2-b1979dd8cbc1","Type":"ContainerDied","Data":"21afa91cf9bb6117a64e51acb7387148118c6b2389c43b53b98420be7c29343d"} Jul 16 22:07:13.884751 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:13.884739 2577 scope.go:117] "RemoveContainer" containerID="21afa91cf9bb6117a64e51acb7387148118c6b2389c43b53b98420be7c29343d" Jul 16 22:07:13.884924 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:07:13.884910 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl_odh-ai-gateway-infra(ba94b701-3810-4a34-b5c2-b1979dd8cbc1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" podUID="ba94b701-3810-4a34-b5c2-b1979dd8cbc1" Jul 16 22:07:13.894094 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:13.894078 2577 scope.go:117] "RemoveContainer" containerID="f0787cf889d747c5f94be1fc8443b5dce0d245d161efca2f2d5e98cdba1bad82" Jul 16 22:07:13.912911 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:13.912889 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft"] Jul 16 22:07:13.920672 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:13.920645 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl"] Jul 16 22:07:15.049053 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.049032 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" Jul 16 22:07:15.051054 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.051037 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" Jul 16 22:07:15.151573 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.151480 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba94b701-3810-4a34-b5c2-b1979dd8cbc1-maas-api-service-ca\") pod \"ba94b701-3810-4a34-b5c2-b1979dd8cbc1\" (UID: \"ba94b701-3810-4a34-b5c2-b1979dd8cbc1\") " Jul 16 22:07:15.151711 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.151598 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b53cd43-a6ad-4b93-a065-3aaf73e4935e-maas-api-service-ca\") pod \"1b53cd43-a6ad-4b93-a065-3aaf73e4935e\" (UID: \"1b53cd43-a6ad-4b93-a065-3aaf73e4935e\") " Jul 16 22:07:15.151898 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.151872 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba94b701-3810-4a34-b5c2-b1979dd8cbc1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba94b701-3810-4a34-b5c2-b1979dd8cbc1" (UID: "ba94b701-3810-4a34-b5c2-b1979dd8cbc1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:07:15.151943 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.151923 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b53cd43-a6ad-4b93-a065-3aaf73e4935e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b53cd43-a6ad-4b93-a065-3aaf73e4935e" (UID: "1b53cd43-a6ad-4b93-a065-3aaf73e4935e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:07:15.252562 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.252524 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba94b701-3810-4a34-b5c2-b1979dd8cbc1-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:07:15.252562 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.252553 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b53cd43-a6ad-4b93-a065-3aaf73e4935e-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:07:15.899437 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.899404 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" Jul 16 22:07:15.899437 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.899428 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft" event={"ID":"1b53cd43-a6ad-4b93-a065-3aaf73e4935e","Type":"ContainerDied","Data":"8cbec4bbac7da73e120c528d70318ef295fb96e0fe15b371dda47de8848841a7"} Jul 16 22:07:15.899726 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.899494 2577 scope.go:117] "RemoveContainer" containerID="0581478a4ce20b5b4004995ede877b1daa435da79c726f6062ca9835124b68ed" Jul 16 22:07:15.901027 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.901009 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" Jul 16 22:07:15.901135 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.901060 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl" event={"ID":"ba94b701-3810-4a34-b5c2-b1979dd8cbc1","Type":"ContainerDied","Data":"5f5c659acdc3c7d1f6b94a6a734e6489b705cc395e360c251825b1878b588572"} Jul 16 22:07:15.908973 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.908930 2577 scope.go:117] "RemoveContainer" containerID="21afa91cf9bb6117a64e51acb7387148118c6b2389c43b53b98420be7c29343d" Jul 16 22:07:15.928178 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.928149 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft"] Jul 16 22:07:15.935083 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.935057 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-tvmft"] Jul 16 22:07:15.949327 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.949298 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl"] Jul 16 22:07:15.953835 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:15.953808 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rl2cl"] Jul 16 22:07:16.450992 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:16.450951 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b53cd43-a6ad-4b93-a065-3aaf73e4935e" path="/var/lib/kubelet/pods/1b53cd43-a6ad-4b93-a065-3aaf73e4935e/volumes" Jul 16 22:07:16.451384 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:16.451331 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba94b701-3810-4a34-b5c2-b1979dd8cbc1" path="/var/lib/kubelet/pods/ba94b701-3810-4a34-b5c2-b1979dd8cbc1/volumes" Jul 16 22:07:17.006621 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.006572 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8"] Jul 16 22:07:17.006817 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.006777 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" podUID="b5c33a47-aba0-456d-a973-9ef9c5471360" containerName="maas-api" containerID="cri-o://e1635d9da59d69dcb93f4756f708009424bdb339db3468c97d41723083cb5ef3" gracePeriod=30 Jul 16 22:07:17.262754 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.262692 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" Jul 16 22:07:17.372238 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.372206 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c33a47-aba0-456d-a973-9ef9c5471360-maas-api-tls\") pod \"b5c33a47-aba0-456d-a973-9ef9c5471360\" (UID: \"b5c33a47-aba0-456d-a973-9ef9c5471360\") " Jul 16 22:07:17.372238 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.372244 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b6bt\" (UniqueName: \"kubernetes.io/projected/b5c33a47-aba0-456d-a973-9ef9c5471360-kube-api-access-8b6bt\") pod \"b5c33a47-aba0-456d-a973-9ef9c5471360\" (UID: \"b5c33a47-aba0-456d-a973-9ef9c5471360\") " Jul 16 22:07:17.372477 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.372314 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c33a47-aba0-456d-a973-9ef9c5471360-openshift-service-ca\") pod \"b5c33a47-aba0-456d-a973-9ef9c5471360\" (UID: \"b5c33a47-aba0-456d-a973-9ef9c5471360\") " Jul 16 22:07:17.372739 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.372713 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c33a47-aba0-456d-a973-9ef9c5471360-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b5c33a47-aba0-456d-a973-9ef9c5471360" (UID: "b5c33a47-aba0-456d-a973-9ef9c5471360"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:07:17.374287 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.374262 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c33a47-aba0-456d-a973-9ef9c5471360-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b5c33a47-aba0-456d-a973-9ef9c5471360" (UID: "b5c33a47-aba0-456d-a973-9ef9c5471360"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:07:17.374468 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.374447 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c33a47-aba0-456d-a973-9ef9c5471360-kube-api-access-8b6bt" (OuterVolumeSpecName: "kube-api-access-8b6bt") pod "b5c33a47-aba0-456d-a973-9ef9c5471360" (UID: "b5c33a47-aba0-456d-a973-9ef9c5471360"). InnerVolumeSpecName "kube-api-access-8b6bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:07:17.472897 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.472860 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c33a47-aba0-456d-a973-9ef9c5471360-maas-api-tls\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:07:17.472897 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.472885 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b6bt\" (UniqueName: \"kubernetes.io/projected/b5c33a47-aba0-456d-a973-9ef9c5471360-kube-api-access-8b6bt\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:07:17.472897 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.472897 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c33a47-aba0-456d-a973-9ef9c5471360-openshift-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:07:17.914672 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.914635 2577 generic.go:356] "Generic (PLEG): container finished" podID="b5c33a47-aba0-456d-a973-9ef9c5471360" containerID="e1635d9da59d69dcb93f4756f708009424bdb339db3468c97d41723083cb5ef3" exitCode=0 Jul 16 22:07:17.914851 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.914684 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" event={"ID":"b5c33a47-aba0-456d-a973-9ef9c5471360","Type":"ContainerDied","Data":"e1635d9da59d69dcb93f4756f708009424bdb339db3468c97d41723083cb5ef3"} Jul 16 22:07:17.914851 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.914691 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" Jul 16 22:07:17.914851 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.914748 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8" event={"ID":"b5c33a47-aba0-456d-a973-9ef9c5471360","Type":"ContainerDied","Data":"03bcb526359fa8d2ea92de6b7c1ce2f21d48326044313a9638e71d59e8504758"} Jul 16 22:07:17.914851 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.914769 2577 scope.go:117] "RemoveContainer" containerID="e1635d9da59d69dcb93f4756f708009424bdb339db3468c97d41723083cb5ef3" Jul 16 22:07:17.923628 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.923604 2577 scope.go:117] "RemoveContainer" containerID="e1635d9da59d69dcb93f4756f708009424bdb339db3468c97d41723083cb5ef3" Jul 16 22:07:17.923865 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:07:17.923849 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1635d9da59d69dcb93f4756f708009424bdb339db3468c97d41723083cb5ef3\": container with ID starting with e1635d9da59d69dcb93f4756f708009424bdb339db3468c97d41723083cb5ef3 not found: ID does not exist" containerID="e1635d9da59d69dcb93f4756f708009424bdb339db3468c97d41723083cb5ef3" Jul 16 22:07:17.923916 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.923873 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1635d9da59d69dcb93f4756f708009424bdb339db3468c97d41723083cb5ef3"} err="failed to get container status \"e1635d9da59d69dcb93f4756f708009424bdb339db3468c97d41723083cb5ef3\": rpc error: code = NotFound desc = could not find container \"e1635d9da59d69dcb93f4756f708009424bdb339db3468c97d41723083cb5ef3\": container with ID starting with e1635d9da59d69dcb93f4756f708009424bdb339db3468c97d41723083cb5ef3 not found: ID does not exist" Jul 16 22:07:17.936838 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.936814 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8"] Jul 16 22:07:17.940578 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:17.940544 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a87ce2-66cb4b9cbc-4zrx8"] Jul 16 22:07:18.450488 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:18.450459 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5c33a47-aba0-456d-a973-9ef9c5471360" path="/var/lib/kubelet/pods/b5c33a47-aba0-456d-a973-9ef9c5471360/volumes" Jul 16 22:07:26.368708 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.368674 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn"] Jul 16 22:07:26.369108 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369024 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba94b701-3810-4a34-b5c2-b1979dd8cbc1" containerName="revoke-keys" Jul 16 22:07:26.369108 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369036 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba94b701-3810-4a34-b5c2-b1979dd8cbc1" containerName="revoke-keys" Jul 16 22:07:26.369108 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369042 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba94b701-3810-4a34-b5c2-b1979dd8cbc1" containerName="revoke-keys" Jul 16 22:07:26.369108 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369048 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba94b701-3810-4a34-b5c2-b1979dd8cbc1" containerName="revoke-keys" Jul 16 22:07:26.369108 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369058 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b53cd43-a6ad-4b93-a065-3aaf73e4935e" containerName="revoke-keys" Jul 16 22:07:26.369108 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369063 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b53cd43-a6ad-4b93-a065-3aaf73e4935e" containerName="revoke-keys" Jul 16 22:07:26.369108 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369075 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5c33a47-aba0-456d-a973-9ef9c5471360" containerName="maas-api" Jul 16 22:07:26.369108 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369080 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5c33a47-aba0-456d-a973-9ef9c5471360" containerName="maas-api" Jul 16 22:07:26.369108 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369086 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba94b701-3810-4a34-b5c2-b1979dd8cbc1" containerName="revoke-keys" Jul 16 22:07:26.369108 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369091 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba94b701-3810-4a34-b5c2-b1979dd8cbc1" containerName="revoke-keys" Jul 16 22:07:26.369108 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369102 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b53cd43-a6ad-4b93-a065-3aaf73e4935e" containerName="revoke-keys" Jul 16 22:07:26.369108 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369108 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b53cd43-a6ad-4b93-a065-3aaf73e4935e" containerName="revoke-keys" Jul 16 22:07:26.369519 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369157 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5c33a47-aba0-456d-a973-9ef9c5471360" containerName="maas-api" Jul 16 22:07:26.369519 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369165 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b53cd43-a6ad-4b93-a065-3aaf73e4935e" containerName="revoke-keys" Jul 16 22:07:26.369519 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369174 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba94b701-3810-4a34-b5c2-b1979dd8cbc1" containerName="revoke-keys" Jul 16 22:07:26.369519 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369179 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b53cd43-a6ad-4b93-a065-3aaf73e4935e" containerName="revoke-keys" Jul 16 22:07:26.369519 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369202 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b53cd43-a6ad-4b93-a065-3aaf73e4935e" containerName="revoke-keys" Jul 16 22:07:26.369519 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369212 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba94b701-3810-4a34-b5c2-b1979dd8cbc1" containerName="revoke-keys" Jul 16 22:07:26.369519 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.369219 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba94b701-3810-4a34-b5c2-b1979dd8cbc1" containerName="revoke-keys" Jul 16 22:07:26.373590 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.373573 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn" Jul 16 22:07:26.376672 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.376647 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 22:07:26.376787 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.376744 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 22:07:26.381696 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.381672 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn"] Jul 16 22:07:26.448104 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.448068 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ddef4e1-1e2c-4ab6-91f8-934a777eccc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn\" (UID: \"8ddef4e1-1e2c-4ab6-91f8-934a777eccc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn" Jul 16 22:07:26.549364 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.549329 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ddef4e1-1e2c-4ab6-91f8-934a777eccc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn\" (UID: \"8ddef4e1-1e2c-4ab6-91f8-934a777eccc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn" Jul 16 22:07:26.549925 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.549901 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ddef4e1-1e2c-4ab6-91f8-934a777eccc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn\" (UID: \"8ddef4e1-1e2c-4ab6-91f8-934a777eccc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn" Jul 16 22:07:26.685832 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.685802 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn" Jul 16 22:07:26.816338 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.816310 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn"] Jul 16 22:07:26.817927 ip-10-0-137-133 kubenswrapper[2577]: W0716 22:07:26.817900 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ddef4e1_1e2c_4ab6_91f8_934a777eccc2.slice/crio-2f5cf7e10292f8a3464040e1831b6b518932512a73ae1d666bc16e793edac6dc WatchSource:0}: Error finding container 2f5cf7e10292f8a3464040e1831b6b518932512a73ae1d666bc16e793edac6dc: Status 404 returned error can't find the container with id 2f5cf7e10292f8a3464040e1831b6b518932512a73ae1d666bc16e793edac6dc Jul 16 22:07:26.962525 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:26.962447 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn" event={"ID":"8ddef4e1-1e2c-4ab6-91f8-934a777eccc2","Type":"ContainerStarted","Data":"2f5cf7e10292f8a3464040e1831b6b518932512a73ae1d666bc16e793edac6dc"} Jul 16 22:07:27.968915 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:27.968882 2577 generic.go:356] "Generic (PLEG): container finished" podID="8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" containerID="dffce93b3776145c5a4c32548e1cff083ccc6d5c7004dece4a2903b4ba2d232b" exitCode=6 Jul 16 22:07:27.969353 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:27.968929 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn" event={"ID":"8ddef4e1-1e2c-4ab6-91f8-934a777eccc2","Type":"ContainerDied","Data":"dffce93b3776145c5a4c32548e1cff083ccc6d5c7004dece4a2903b4ba2d232b"} Jul 16 22:07:27.969353 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:27.969103 2577 scope.go:117] "RemoveContainer" containerID="dffce93b3776145c5a4c32548e1cff083ccc6d5c7004dece4a2903b4ba2d232b" Jul 16 22:07:28.975999 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:28.975964 2577 generic.go:356] "Generic (PLEG): container finished" podID="8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" containerID="ce5f8a4dc22e838d4b3d11d4cd48cd94cfb347faa9b8a9e8f290c1f21373a7a4" exitCode=6 Jul 16 22:07:28.976492 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:28.976021 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn" event={"ID":"8ddef4e1-1e2c-4ab6-91f8-934a777eccc2","Type":"ContainerDied","Data":"ce5f8a4dc22e838d4b3d11d4cd48cd94cfb347faa9b8a9e8f290c1f21373a7a4"} Jul 16 22:07:28.976492 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:28.976073 2577 scope.go:117] "RemoveContainer" containerID="dffce93b3776145c5a4c32548e1cff083ccc6d5c7004dece4a2903b4ba2d232b" Jul 16 22:07:28.976492 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:28.976273 2577 scope.go:117] "RemoveContainer" containerID="ce5f8a4dc22e838d4b3d11d4cd48cd94cfb347faa9b8a9e8f290c1f21373a7a4" Jul 16 22:07:28.976656 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:07:28.976512 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn_odh-ai-gateway-infra(8ddef4e1-1e2c-4ab6-91f8-934a777eccc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn" podUID="8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" Jul 16 22:07:29.983461 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:29.983378 2577 scope.go:117] "RemoveContainer" containerID="ce5f8a4dc22e838d4b3d11d4cd48cd94cfb347faa9b8a9e8f290c1f21373a7a4" Jul 16 22:07:29.983910 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:07:29.983652 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn_odh-ai-gateway-infra(8ddef4e1-1e2c-4ab6-91f8-934a777eccc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn" podUID="8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" Jul 16 22:07:32.252688 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:32.252656 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg"] Jul 16 22:07:32.253167 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:32.252857 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" podUID="bf9dbd76-938f-4670-9d14-1f1f3b44ff85" containerName="istio-proxy" containerID="cri-o://77bdd743395fd88b78395b71028eef5de051ace203e7532c161edf67fb90e5bc" gracePeriod=30 Jul 16 22:07:34.829208 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:34.829153 2577 patch_prober.go:28] interesting pod/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 22:07:34.829575 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:34.829227 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" podUID="bf9dbd76-938f-4670-9d14-1f1f3b44ff85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:07:37.511717 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.511692 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:07:37.648632 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.648601 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-podinfo\") pod \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " Jul 16 22:07:37.648803 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.648652 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-envoy\") pod \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " Jul 16 22:07:37.648803 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.648668 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istiod-ca-cert\") pod \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " Jul 16 22:07:37.648803 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.648687 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-workload-certs\") pod \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " Jul 16 22:07:37.648803 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.648722 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-credential-socket\") pod \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " Jul 16 22:07:37.648803 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.648782 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-data\") pod \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " Jul 16 22:07:37.649080 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.648803 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-workload-socket\") pod \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " Jul 16 22:07:37.649080 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.648833 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-token\") pod \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " Jul 16 22:07:37.649080 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.648861 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxx4m\" (UniqueName: \"kubernetes.io/projected/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-kube-api-access-jxx4m\") pod \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\" (UID: \"bf9dbd76-938f-4670-9d14-1f1f3b44ff85\") " Jul 16 22:07:37.649080 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.649003 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bf9dbd76-938f-4670-9d14-1f1f3b44ff85" (UID: "bf9dbd76-938f-4670-9d14-1f1f3b44ff85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:07:37.649080 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.649055 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bf9dbd76-938f-4670-9d14-1f1f3b44ff85" (UID: "bf9dbd76-938f-4670-9d14-1f1f3b44ff85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:07:37.649373 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.649100 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bf9dbd76-938f-4670-9d14-1f1f3b44ff85" (UID: "bf9dbd76-938f-4670-9d14-1f1f3b44ff85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:07:37.649373 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.649112 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bf9dbd76-938f-4670-9d14-1f1f3b44ff85" (UID: "bf9dbd76-938f-4670-9d14-1f1f3b44ff85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:07:37.649373 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.649224 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-workload-socket\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:07:37.649373 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.649243 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istiod-ca-cert\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:07:37.649373 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.649257 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-workload-certs\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:07:37.649373 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.649269 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-credential-socket\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:07:37.649747 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.649720 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-data" (OuterVolumeSpecName: "istio-data") pod "bf9dbd76-938f-4670-9d14-1f1f3b44ff85" (UID: "bf9dbd76-938f-4670-9d14-1f1f3b44ff85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:07:37.650926 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.650900 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bf9dbd76-938f-4670-9d14-1f1f3b44ff85" (UID: "bf9dbd76-938f-4670-9d14-1f1f3b44ff85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:07:37.651091 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.651076 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bf9dbd76-938f-4670-9d14-1f1f3b44ff85" (UID: "bf9dbd76-938f-4670-9d14-1f1f3b44ff85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 22:07:37.651484 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.651467 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-token" (OuterVolumeSpecName: "istio-token") pod "bf9dbd76-938f-4670-9d14-1f1f3b44ff85" (UID: "bf9dbd76-938f-4670-9d14-1f1f3b44ff85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:07:37.651556 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.651480 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-kube-api-access-jxx4m" (OuterVolumeSpecName: "kube-api-access-jxx4m") pod "bf9dbd76-938f-4670-9d14-1f1f3b44ff85" (UID: "bf9dbd76-938f-4670-9d14-1f1f3b44ff85"). InnerVolumeSpecName "kube-api-access-jxx4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:07:37.750283 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.750247 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-envoy\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:07:37.750283 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.750278 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-data\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:07:37.750283 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.750290 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-token\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:07:37.750565 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.750303 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxx4m\" (UniqueName: \"kubernetes.io/projected/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-kube-api-access-jxx4m\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:07:37.750565 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:37.750317 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9dbd76-938f-4670-9d14-1f1f3b44ff85-istio-podinfo\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:07:38.028018 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:38.027929 2577 generic.go:356] "Generic (PLEG): container finished" podID="bf9dbd76-938f-4670-9d14-1f1f3b44ff85" containerID="77bdd743395fd88b78395b71028eef5de051ace203e7532c161edf67fb90e5bc" exitCode=0 Jul 16 22:07:38.028018 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:38.027970 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" Jul 16 22:07:38.028228 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:38.028021 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" event={"ID":"bf9dbd76-938f-4670-9d14-1f1f3b44ff85","Type":"ContainerDied","Data":"77bdd743395fd88b78395b71028eef5de051ace203e7532c161edf67fb90e5bc"} Jul 16 22:07:38.028228 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:38.028073 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg" event={"ID":"bf9dbd76-938f-4670-9d14-1f1f3b44ff85","Type":"ContainerDied","Data":"0fbec1b9c17d6ebd31259a796b9f14cd5a733ce4b1d43cd84e83a378de92b38d"} Jul 16 22:07:38.028228 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:38.028103 2577 scope.go:117] "RemoveContainer" containerID="77bdd743395fd88b78395b71028eef5de051ace203e7532c161edf67fb90e5bc" Jul 16 22:07:38.037653 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:38.037635 2577 scope.go:117] "RemoveContainer" containerID="be8413ed3c47c5a7569a7575a8180949e67d8392ba2f4e16d4eb8c9a83e9c4cf" Jul 16 22:07:38.045628 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:38.045610 2577 scope.go:117] "RemoveContainer" containerID="77bdd743395fd88b78395b71028eef5de051ace203e7532c161edf67fb90e5bc" Jul 16 22:07:38.045861 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:07:38.045842 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77bdd743395fd88b78395b71028eef5de051ace203e7532c161edf67fb90e5bc\": container with ID starting with 77bdd743395fd88b78395b71028eef5de051ace203e7532c161edf67fb90e5bc not found: ID does not exist" containerID="77bdd743395fd88b78395b71028eef5de051ace203e7532c161edf67fb90e5bc" Jul 16 22:07:38.045940 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:38.045873 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77bdd743395fd88b78395b71028eef5de051ace203e7532c161edf67fb90e5bc"} err="failed to get container status \"77bdd743395fd88b78395b71028eef5de051ace203e7532c161edf67fb90e5bc\": rpc error: code = NotFound desc = could not find container \"77bdd743395fd88b78395b71028eef5de051ace203e7532c161edf67fb90e5bc\": container with ID starting with 77bdd743395fd88b78395b71028eef5de051ace203e7532c161edf67fb90e5bc not found: ID does not exist" Jul 16 22:07:38.045940 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:38.045898 2577 scope.go:117] "RemoveContainer" containerID="be8413ed3c47c5a7569a7575a8180949e67d8392ba2f4e16d4eb8c9a83e9c4cf" Jul 16 22:07:38.046117 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:07:38.046101 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be8413ed3c47c5a7569a7575a8180949e67d8392ba2f4e16d4eb8c9a83e9c4cf\": container with ID starting with be8413ed3c47c5a7569a7575a8180949e67d8392ba2f4e16d4eb8c9a83e9c4cf not found: ID does not exist" containerID="be8413ed3c47c5a7569a7575a8180949e67d8392ba2f4e16d4eb8c9a83e9c4cf" Jul 16 22:07:38.046160 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:38.046122 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8413ed3c47c5a7569a7575a8180949e67d8392ba2f4e16d4eb8c9a83e9c4cf"} err="failed to get container status \"be8413ed3c47c5a7569a7575a8180949e67d8392ba2f4e16d4eb8c9a83e9c4cf\": rpc error: code = NotFound desc = could not find container \"be8413ed3c47c5a7569a7575a8180949e67d8392ba2f4e16d4eb8c9a83e9c4cf\": container with ID starting with be8413ed3c47c5a7569a7575a8180949e67d8392ba2f4e16d4eb8c9a83e9c4cf not found: ID does not exist" Jul 16 22:07:38.054999 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:38.054975 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg"] Jul 16 22:07:38.061808 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:38.061783 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-a87ce2-openshift-default-5b845d97f-rnhlg"] Jul 16 22:07:38.451051 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:38.451016 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf9dbd76-938f-4670-9d14-1f1f3b44ff85" path="/var/lib/kubelet/pods/bf9dbd76-938f-4670-9d14-1f1f3b44ff85/volumes" Jul 16 22:07:38.911149 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:38.911121 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9f7d7dc89-6jjxn_f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3/copy-manifests/0.log" Jul 16 22:07:38.919309 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:38.919283 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9f7d7dc89-6jjxn_f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3/manager/0.log" Jul 16 22:07:39.597750 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:39.597718 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737305-m79dl_3b80877d-921d-42d8-a8fa-74d0826123c3/cleanup/0.log" Jul 16 22:07:39.717530 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:39.717500 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737320-dz2bj_f62dc075-1101-4952-a656-339b5d148750/cleanup/0.log" Jul 16 22:07:39.831071 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:39.831043 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn_8ddef4e1-1e2c-4ab6-91f8-934a777eccc2/revoke-keys/1.log" Jul 16 22:07:40.370696 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:40.370666 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-6e4744-tdzjs_b279c32c-76c4-42d4-98c5-ce5f662acd45/revoke-keys/0.log" Jul 16 22:07:40.587418 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:40.587389 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-f8eae57b-ct5nr_4059e66f-188c-4ae4-a47b-dc575fe02494/revoke-keys/0.log" Jul 16 22:07:41.663102 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:41.663069 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw_f2cbded0-5f0e-4865-a7d5-cd2619506273/pull/0.log" Jul 16 22:07:41.670117 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:41.670090 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw_f2cbded0-5f0e-4865-a7d5-cd2619506273/extract/0.log" Jul 16 22:07:41.676264 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:41.676247 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw_f2cbded0-5f0e-4865-a7d5-cd2619506273/util/0.log" Jul 16 22:07:41.785590 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:41.785559 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs_8ee8cd36-2859-4bf3-a453-e468eb83048d/extract/0.log" Jul 16 22:07:41.792078 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:41.792053 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs_8ee8cd36-2859-4bf3-a453-e468eb83048d/util/0.log" Jul 16 22:07:41.799274 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:41.799249 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs_8ee8cd36-2859-4bf3-a453-e468eb83048d/pull/0.log" Jul 16 22:07:41.903714 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:41.903680 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5_6ec078ff-b939-4e9a-a206-81ea686b278b/util/0.log" Jul 16 22:07:41.910358 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:41.910332 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5_6ec078ff-b939-4e9a-a206-81ea686b278b/pull/0.log" Jul 16 22:07:41.916852 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:41.916770 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5_6ec078ff-b939-4e9a-a206-81ea686b278b/extract/0.log" Jul 16 22:07:42.026515 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:42.026481 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr_92841e70-433a-42e1-8313-5419ab6f6399/util/0.log" Jul 16 22:07:42.032877 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:42.032842 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr_92841e70-433a-42e1-8313-5419ab6f6399/pull/0.log" Jul 16 22:07:42.039361 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:42.039330 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr_92841e70-433a-42e1-8313-5419ab6f6399/extract/0.log" Jul 16 22:07:43.071100 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:43.071066 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-7khc2_3c42ddfe-8200-4594-8c0f-12b3233d00e1/manager/0.log" Jul 16 22:07:43.432675 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:43.432643 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4pfstf_eb922b04-96b8-401f-8610-017771c0b3b8/istio-proxy/0.log" Jul 16 22:07:43.445165 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:43.445144 2577 scope.go:117] "RemoveContainer" containerID="ce5f8a4dc22e838d4b3d11d4cd48cd94cfb347faa9b8a9e8f290c1f21373a7a4" Jul 16 22:07:43.550194 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:43.550147 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-585pl_42846e1e-6fd3-4bc3-9ec2-c558feb88415/discovery/0.log" Jul 16 22:07:44.067141 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:44.067108 2577 generic.go:356] "Generic (PLEG): container finished" podID="8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" containerID="13573db95342f5010065819fe3df6457c129714ba18c06fc4214198ba6134fb3" exitCode=6 Jul 16 22:07:44.067312 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:44.067192 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn" event={"ID":"8ddef4e1-1e2c-4ab6-91f8-934a777eccc2","Type":"ContainerDied","Data":"13573db95342f5010065819fe3df6457c129714ba18c06fc4214198ba6134fb3"} Jul 16 22:07:44.067312 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:44.067257 2577 scope.go:117] "RemoveContainer" containerID="ce5f8a4dc22e838d4b3d11d4cd48cd94cfb347faa9b8a9e8f290c1f21373a7a4" Jul 16 22:07:44.067578 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:44.067557 2577 scope.go:117] "RemoveContainer" containerID="13573db95342f5010065819fe3df6457c129714ba18c06fc4214198ba6134fb3" Jul 16 22:07:44.067826 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:07:44.067799 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn_odh-ai-gateway-infra(8ddef4e1-1e2c-4ab6-91f8-934a777eccc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn" podUID="8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" Jul 16 22:07:44.710386 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:44.710354 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-jq76f_0cb7eca7-64e2-4ab2-a119-053ccf18c111/main/0.log" Jul 16 22:07:45.111116 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:45.111080 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn"] Jul 16 22:07:45.242226 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:45.242204 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn" Jul 16 22:07:45.319765 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:45.319673 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ddef4e1-1e2c-4ab6-91f8-934a777eccc2-maas-api-service-ca\") pod \"8ddef4e1-1e2c-4ab6-91f8-934a777eccc2\" (UID: \"8ddef4e1-1e2c-4ab6-91f8-934a777eccc2\") " Jul 16 22:07:45.320053 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:45.320035 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ddef4e1-1e2c-4ab6-91f8-934a777eccc2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" (UID: "8ddef4e1-1e2c-4ab6-91f8-934a777eccc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:07:45.420344 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:45.420309 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ddef4e1-1e2c-4ab6-91f8-934a777eccc2-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:07:46.080493 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.080458 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn" event={"ID":"8ddef4e1-1e2c-4ab6-91f8-934a777eccc2","Type":"ContainerDied","Data":"2f5cf7e10292f8a3464040e1831b6b518932512a73ae1d666bc16e793edac6dc"} Jul 16 22:07:46.080493 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.080479 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn" Jul 16 22:07:46.081000 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.080526 2577 scope.go:117] "RemoveContainer" containerID="13573db95342f5010065819fe3df6457c129714ba18c06fc4214198ba6134fb3" Jul 16 22:07:46.104965 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.104938 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn"] Jul 16 22:07:46.109689 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.109663 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f60157c3-rmwnn"] Jul 16 22:07:46.457050 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.457015 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" path="/var/lib/kubelet/pods/8ddef4e1-1e2c-4ab6-91f8-934a777eccc2/volumes" Jul 16 22:07:46.715626 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.715547 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2"] Jul 16 22:07:46.715882 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.715871 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" containerName="revoke-keys" Jul 16 22:07:46.715925 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.715884 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" containerName="revoke-keys" Jul 16 22:07:46.715925 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.715891 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf9dbd76-938f-4670-9d14-1f1f3b44ff85" containerName="istio-proxy" Jul 16 22:07:46.715925 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.715897 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf9dbd76-938f-4670-9d14-1f1f3b44ff85" containerName="istio-proxy" Jul 16 22:07:46.715925 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.715905 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" containerName="revoke-keys" Jul 16 22:07:46.715925 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.715910 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" containerName="revoke-keys" Jul 16 22:07:46.715925 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.715918 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" containerName="revoke-keys" Jul 16 22:07:46.715925 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.715923 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" containerName="revoke-keys" Jul 16 22:07:46.716142 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.715933 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf9dbd76-938f-4670-9d14-1f1f3b44ff85" containerName="istio-proxy" Jul 16 22:07:46.716142 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.715940 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf9dbd76-938f-4670-9d14-1f1f3b44ff85" containerName="istio-proxy" Jul 16 22:07:46.716142 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.715949 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b53cd43-a6ad-4b93-a065-3aaf73e4935e" containerName="revoke-keys" Jul 16 22:07:46.716142 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.715954 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b53cd43-a6ad-4b93-a065-3aaf73e4935e" containerName="revoke-keys" Jul 16 22:07:46.716142 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.716003 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" containerName="revoke-keys" Jul 16 22:07:46.716142 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.716011 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf9dbd76-938f-4670-9d14-1f1f3b44ff85" containerName="istio-proxy" Jul 16 22:07:46.716142 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.716017 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" containerName="revoke-keys" Jul 16 22:07:46.716142 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.716022 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf9dbd76-938f-4670-9d14-1f1f3b44ff85" containerName="istio-proxy" Jul 16 22:07:46.720506 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.720489 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" Jul 16 22:07:46.723504 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.723477 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 22:07:46.723621 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.723571 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46j7\"" Jul 16 22:07:46.728434 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.727992 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2"] Jul 16 22:07:46.728596 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.728570 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e53153-a08e-43f9-abf9-e23473a54c04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2\" (UID: \"e7e53153-a08e-43f9-abf9-e23473a54c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" Jul 16 22:07:46.829268 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.829223 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e53153-a08e-43f9-abf9-e23473a54c04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2\" (UID: \"e7e53153-a08e-43f9-abf9-e23473a54c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" Jul 16 22:07:46.829862 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:46.829841 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e53153-a08e-43f9-abf9-e23473a54c04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2\" (UID: \"e7e53153-a08e-43f9-abf9-e23473a54c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" Jul 16 22:07:47.032482 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:47.032399 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" Jul 16 22:07:47.169047 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:47.169018 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2"] Jul 16 22:07:47.170862 ip-10-0-137-133 kubenswrapper[2577]: W0716 22:07:47.170827 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e53153_a08e_43f9_abf9_e23473a54c04.slice/crio-53396df61073d0236647fee66b4db6868a16a895201d37d1d1e76c520ba92a05 WatchSource:0}: Error finding container 53396df61073d0236647fee66b4db6868a16a895201d37d1d1e76c520ba92a05: Status 404 returned error can't find the container with id 53396df61073d0236647fee66b4db6868a16a895201d37d1d1e76c520ba92a05 Jul 16 22:07:47.953296 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:47.953256 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6"] Jul 16 22:07:47.953723 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:47.953704 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ddef4e1-1e2c-4ab6-91f8-934a777eccc2" containerName="revoke-keys" Jul 16 22:07:47.956679 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:47.956656 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" Jul 16 22:07:47.966291 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:47.966259 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6"] Jul 16 22:07:48.039034 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:48.038957 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4e257c-afc4-4e28-a2fd-a1ab1de74638-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6\" (UID: \"de4e257c-afc4-4e28-a2fd-a1ab1de74638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" Jul 16 22:07:48.098263 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:48.098231 2577 generic.go:356] "Generic (PLEG): container finished" podID="e7e53153-a08e-43f9-abf9-e23473a54c04" containerID="7c5875f79887e77b7027ec462f694f1f7a70983399f5f3d206203c55e7fa1d7a" exitCode=6 Jul 16 22:07:48.098263 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:48.098267 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" event={"ID":"e7e53153-a08e-43f9-abf9-e23473a54c04","Type":"ContainerDied","Data":"7c5875f79887e77b7027ec462f694f1f7a70983399f5f3d206203c55e7fa1d7a"} Jul 16 22:07:48.098462 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:48.098303 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" event={"ID":"e7e53153-a08e-43f9-abf9-e23473a54c04","Type":"ContainerStarted","Data":"53396df61073d0236647fee66b4db6868a16a895201d37d1d1e76c520ba92a05"} Jul 16 22:07:48.098505 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:48.098481 2577 scope.go:117] "RemoveContainer" containerID="7c5875f79887e77b7027ec462f694f1f7a70983399f5f3d206203c55e7fa1d7a" Jul 16 22:07:48.140163 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:48.140133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4e257c-afc4-4e28-a2fd-a1ab1de74638-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6\" (UID: \"de4e257c-afc4-4e28-a2fd-a1ab1de74638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" Jul 16 22:07:48.140680 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:48.140663 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4e257c-afc4-4e28-a2fd-a1ab1de74638-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6\" (UID: \"de4e257c-afc4-4e28-a2fd-a1ab1de74638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" Jul 16 22:07:48.268001 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:48.267964 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" Jul 16 22:07:48.402359 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:48.402336 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6"] Jul 16 22:07:48.404226 ip-10-0-137-133 kubenswrapper[2577]: W0716 22:07:48.404174 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4e257c_afc4_4e28_a2fd_a1ab1de74638.slice/crio-98bb158ac1a2bc3470a9b33541d7dae014e4dd2acda164ed02a27de5f52542f3 WatchSource:0}: Error finding container 98bb158ac1a2bc3470a9b33541d7dae014e4dd2acda164ed02a27de5f52542f3: Status 404 returned error can't find the container with id 98bb158ac1a2bc3470a9b33541d7dae014e4dd2acda164ed02a27de5f52542f3 Jul 16 22:07:49.105675 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:49.105648 2577 generic.go:356] "Generic (PLEG): container finished" podID="e7e53153-a08e-43f9-abf9-e23473a54c04" containerID="cec7f6d652581aff5b7ab079fb16c03e3d26eb175a2ad24fae06d502cf03af08" exitCode=6 Jul 16 22:07:49.105834 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:49.105740 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" event={"ID":"e7e53153-a08e-43f9-abf9-e23473a54c04","Type":"ContainerDied","Data":"cec7f6d652581aff5b7ab079fb16c03e3d26eb175a2ad24fae06d502cf03af08"} Jul 16 22:07:49.105834 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:49.105802 2577 scope.go:117] "RemoveContainer" containerID="7c5875f79887e77b7027ec462f694f1f7a70983399f5f3d206203c55e7fa1d7a" Jul 16 22:07:49.105959 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:49.105938 2577 scope.go:117] "RemoveContainer" containerID="cec7f6d652581aff5b7ab079fb16c03e3d26eb175a2ad24fae06d502cf03af08" Jul 16 22:07:49.106196 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:07:49.106157 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2_odh-ai-gateway-infra(e7e53153-a08e-43f9-abf9-e23473a54c04)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" podUID="e7e53153-a08e-43f9-abf9-e23473a54c04" Jul 16 22:07:49.107264 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:49.107245 2577 generic.go:356] "Generic (PLEG): container finished" podID="de4e257c-afc4-4e28-a2fd-a1ab1de74638" containerID="57f0b0080835a7dea25139417c276c33cc09d44f0cc35f1bd886456e4bfdaef6" exitCode=6 Jul 16 22:07:49.107361 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:49.107305 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" event={"ID":"de4e257c-afc4-4e28-a2fd-a1ab1de74638","Type":"ContainerDied","Data":"57f0b0080835a7dea25139417c276c33cc09d44f0cc35f1bd886456e4bfdaef6"} Jul 16 22:07:49.107361 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:49.107333 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" event={"ID":"de4e257c-afc4-4e28-a2fd-a1ab1de74638","Type":"ContainerStarted","Data":"98bb158ac1a2bc3470a9b33541d7dae014e4dd2acda164ed02a27de5f52542f3"} Jul 16 22:07:49.107456 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:49.107444 2577 scope.go:117] "RemoveContainer" containerID="57f0b0080835a7dea25139417c276c33cc09d44f0cc35f1bd886456e4bfdaef6" Jul 16 22:07:50.115116 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.115015 2577 scope.go:117] "RemoveContainer" containerID="cec7f6d652581aff5b7ab079fb16c03e3d26eb175a2ad24fae06d502cf03af08" Jul 16 22:07:50.115572 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:07:50.115255 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2_odh-ai-gateway-infra(e7e53153-a08e-43f9-abf9-e23473a54c04)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" podUID="e7e53153-a08e-43f9-abf9-e23473a54c04" Jul 16 22:07:50.116613 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.116588 2577 generic.go:356] "Generic (PLEG): container finished" podID="de4e257c-afc4-4e28-a2fd-a1ab1de74638" containerID="02f0170a98621ba4bc3ecac71575f698718f95dcbbb45ecb29cca7cab7eaa770" exitCode=6 Jul 16 22:07:50.116751 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.116634 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" event={"ID":"de4e257c-afc4-4e28-a2fd-a1ab1de74638","Type":"ContainerDied","Data":"02f0170a98621ba4bc3ecac71575f698718f95dcbbb45ecb29cca7cab7eaa770"} Jul 16 22:07:50.116751 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.116688 2577 scope.go:117] "RemoveContainer" containerID="57f0b0080835a7dea25139417c276c33cc09d44f0cc35f1bd886456e4bfdaef6" Jul 16 22:07:50.116914 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.116900 2577 scope.go:117] "RemoveContainer" containerID="02f0170a98621ba4bc3ecac71575f698718f95dcbbb45ecb29cca7cab7eaa770" Jul 16 22:07:50.117083 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:07:50.117062 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6_odh-ai-gateway-infra(de4e257c-afc4-4e28-a2fd-a1ab1de74638)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" podUID="de4e257c-afc4-4e28-a2fd-a1ab1de74638" Jul 16 22:07:50.521513 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.521479 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bq49w/must-gather-j97k9"] Jul 16 22:07:50.526315 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.526291 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq49w/must-gather-j97k9" Jul 16 22:07:50.530447 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.530423 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bq49w\"/\"kube-root-ca.crt\"" Jul 16 22:07:50.531166 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.531151 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bq49w\"/\"openshift-service-ca.crt\"" Jul 16 22:07:50.542521 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.542499 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq49w/must-gather-j97k9"] Jul 16 22:07:50.561248 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.561223 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92f5d2e8-1f25-499e-99e4-9c4d0672613b-must-gather-output\") pod \"must-gather-j97k9\" (UID: \"92f5d2e8-1f25-499e-99e4-9c4d0672613b\") " pod="openshift-must-gather-bq49w/must-gather-j97k9" Jul 16 22:07:50.561389 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.561320 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxmlr\" (UniqueName: \"kubernetes.io/projected/92f5d2e8-1f25-499e-99e4-9c4d0672613b-kube-api-access-zxmlr\") pod \"must-gather-j97k9\" (UID: \"92f5d2e8-1f25-499e-99e4-9c4d0672613b\") " pod="openshift-must-gather-bq49w/must-gather-j97k9" Jul 16 22:07:50.661847 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.661814 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxmlr\" (UniqueName: \"kubernetes.io/projected/92f5d2e8-1f25-499e-99e4-9c4d0672613b-kube-api-access-zxmlr\") pod \"must-gather-j97k9\" (UID: \"92f5d2e8-1f25-499e-99e4-9c4d0672613b\") " pod="openshift-must-gather-bq49w/must-gather-j97k9" Jul 16 22:07:50.661988 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.661863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92f5d2e8-1f25-499e-99e4-9c4d0672613b-must-gather-output\") pod \"must-gather-j97k9\" (UID: \"92f5d2e8-1f25-499e-99e4-9c4d0672613b\") " pod="openshift-must-gather-bq49w/must-gather-j97k9" Jul 16 22:07:50.662174 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.662160 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92f5d2e8-1f25-499e-99e4-9c4d0672613b-must-gather-output\") pod \"must-gather-j97k9\" (UID: \"92f5d2e8-1f25-499e-99e4-9c4d0672613b\") " pod="openshift-must-gather-bq49w/must-gather-j97k9" Jul 16 22:07:50.670941 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.670922 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxmlr\" (UniqueName: \"kubernetes.io/projected/92f5d2e8-1f25-499e-99e4-9c4d0672613b-kube-api-access-zxmlr\") pod \"must-gather-j97k9\" (UID: \"92f5d2e8-1f25-499e-99e4-9c4d0672613b\") " pod="openshift-must-gather-bq49w/must-gather-j97k9" Jul 16 22:07:50.835013 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.834920 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq49w/must-gather-j97k9" Jul 16 22:07:50.999151 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:50.999127 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq49w/must-gather-j97k9"] Jul 16 22:07:51.000398 ip-10-0-137-133 kubenswrapper[2577]: W0716 22:07:51.000369 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92f5d2e8_1f25_499e_99e4_9c4d0672613b.slice/crio-1faf393658057bf3bd7131515839f6036ed8e418923cfc7be5969d21adfb1257 WatchSource:0}: Error finding container 1faf393658057bf3bd7131515839f6036ed8e418923cfc7be5969d21adfb1257: Status 404 returned error can't find the container with id 1faf393658057bf3bd7131515839f6036ed8e418923cfc7be5969d21adfb1257 Jul 16 22:07:51.002394 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:51.002374 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 22:07:51.123414 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:51.123336 2577 scope.go:117] "RemoveContainer" containerID="02f0170a98621ba4bc3ecac71575f698718f95dcbbb45ecb29cca7cab7eaa770" Jul 16 22:07:51.123832 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:07:51.123579 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6_odh-ai-gateway-infra(de4e257c-afc4-4e28-a2fd-a1ab1de74638)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" podUID="de4e257c-afc4-4e28-a2fd-a1ab1de74638" Jul 16 22:07:51.124231 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:51.124213 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq49w/must-gather-j97k9" event={"ID":"92f5d2e8-1f25-499e-99e4-9c4d0672613b","Type":"ContainerStarted","Data":"1faf393658057bf3bd7131515839f6036ed8e418923cfc7be5969d21adfb1257"} Jul 16 22:07:53.139024 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:53.138982 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq49w/must-gather-j97k9" event={"ID":"92f5d2e8-1f25-499e-99e4-9c4d0672613b","Type":"ContainerStarted","Data":"777d38c848568e03d07fb056a044d4cdad979962986bc6cf45c4231482f4067a"} Jul 16 22:07:53.139521 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:53.139041 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq49w/must-gather-j97k9" event={"ID":"92f5d2e8-1f25-499e-99e4-9c4d0672613b","Type":"ContainerStarted","Data":"4e9b7e5dc4a7b083c68cf91ba696908ebd89210786fb5ea56b64b4ac954155f5"} Jul 16 22:07:53.158000 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:53.157947 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bq49w/must-gather-j97k9" podStartSLOduration=1.511461401 podStartE2EDuration="3.157930209s" totalImagesPullingTime="1.646468808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:07:53.157126406 +0000 UTC m=+3489.332234602" watchObservedRunningTime="2026-07-16 22:07:53.157930209 +0000 UTC m=+3489.333038401" Jul 16 22:07:54.330299 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:54.330260 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2xbp6_3bd19c30-8019-4896-aa14-62214c3e042c/global-pull-secret-syncer/0.log" Jul 16 22:07:54.579420 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:54.579381 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s2pdw_21f02b78-840d-4d87-a3da-ef3a063424a2/konnectivity-agent/0.log" Jul 16 22:07:54.661885 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:54.661847 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-133.ec2.internal_cd1e9b97c4c8ceff5fa6e561d0396716/haproxy/0.log" Jul 16 22:07:59.333831 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:59.333800 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw_f2cbded0-5f0e-4865-a7d5-cd2619506273/extract/0.log" Jul 16 22:07:59.374673 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:59.374642 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw_f2cbded0-5f0e-4865-a7d5-cd2619506273/util/0.log" Jul 16 22:07:59.434338 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:59.434305 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tn6fw_f2cbded0-5f0e-4865-a7d5-cd2619506273/pull/0.log" Jul 16 22:07:59.505819 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:59.505795 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs_8ee8cd36-2859-4bf3-a453-e468eb83048d/extract/0.log" Jul 16 22:07:59.554058 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:59.554019 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs_8ee8cd36-2859-4bf3-a453-e468eb83048d/util/0.log" Jul 16 22:07:59.608212 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:59.608109 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06xrhs_8ee8cd36-2859-4bf3-a453-e468eb83048d/pull/0.log" Jul 16 22:07:59.641136 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:59.641107 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5_6ec078ff-b939-4e9a-a206-81ea686b278b/extract/0.log" Jul 16 22:07:59.666154 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:59.666129 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5_6ec078ff-b939-4e9a-a206-81ea686b278b/util/0.log" Jul 16 22:07:59.697721 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:59.697695 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dn6n5_6ec078ff-b939-4e9a-a206-81ea686b278b/pull/0.log" Jul 16 22:07:59.731570 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:59.731530 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr_92841e70-433a-42e1-8313-5419ab6f6399/extract/0.log" Jul 16 22:07:59.759020 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:59.758983 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr_92841e70-433a-42e1-8313-5419ab6f6399/util/0.log" Jul 16 22:07:59.781304 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:07:59.781258 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxfqr_92841e70-433a-42e1-8313-5419ab6f6399/pull/0.log" Jul 16 22:08:00.616611 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:00.616573 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-7khc2_3c42ddfe-8200-4594-8c0f-12b3233d00e1/manager/0.log" Jul 16 22:08:01.445961 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:01.445926 2577 scope.go:117] "RemoveContainer" containerID="cec7f6d652581aff5b7ab079fb16c03e3d26eb175a2ad24fae06d502cf03af08" Jul 16 22:08:02.223204 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.222360 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" event={"ID":"e7e53153-a08e-43f9-abf9-e23473a54c04","Type":"ContainerStarted","Data":"2e31249c1277c4d3453b46d2ffc6e7164fb23ac4f4bdfe1e06147f5e281b1dad"} Jul 16 22:08:02.243273 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.243205 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" podStartSLOduration=16.243170859 podStartE2EDuration="16.243170859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:08:02.240997617 +0000 UTC m=+3498.416105810" watchObservedRunningTime="2026-07-16 22:08:02.243170859 +0000 UTC m=+3498.418279053" Jul 16 22:08:02.281115 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.281088 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fecc5f4c-26b0-420e-9c65-1bcba45fb17f/alertmanager/0.log" Jul 16 22:08:02.311263 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.311231 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fecc5f4c-26b0-420e-9c65-1bcba45fb17f/config-reloader/0.log" Jul 16 22:08:02.336614 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.336584 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fecc5f4c-26b0-420e-9c65-1bcba45fb17f/kube-rbac-proxy-web/0.log" Jul 16 22:08:02.375343 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.375312 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fecc5f4c-26b0-420e-9c65-1bcba45fb17f/kube-rbac-proxy/0.log" Jul 16 22:08:02.402410 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.402376 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fecc5f4c-26b0-420e-9c65-1bcba45fb17f/kube-rbac-proxy-metric/0.log" Jul 16 22:08:02.427511 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.427482 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fecc5f4c-26b0-420e-9c65-1bcba45fb17f/prom-label-proxy/0.log" Jul 16 22:08:02.458909 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.458884 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fecc5f4c-26b0-420e-9c65-1bcba45fb17f/init-config-reloader/0.log" Jul 16 22:08:02.568111 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.568082 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-mqs5r_764206ea-3f09-45a2-aae6-47dd7908d230/kube-state-metrics/0.log" Jul 16 22:08:02.589849 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.589822 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-mqs5r_764206ea-3f09-45a2-aae6-47dd7908d230/kube-rbac-proxy-main/0.log" Jul 16 22:08:02.616204 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.616114 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-mqs5r_764206ea-3f09-45a2-aae6-47dd7908d230/kube-rbac-proxy-self/0.log" Jul 16 22:08:02.646078 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.646042 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7479cf6c97-59v8t_202b3f86-d79e-457b-abd8-9fe67f18fb69/metrics-server/0.log" Jul 16 22:08:02.672956 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.672927 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-gddmg_e4bb1fec-1520-4f19-ba3d-c95eb209e8dd/monitoring-plugin/0.log" Jul 16 22:08:02.799762 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.799681 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-klcvk_a36431c2-8598-4a45-b16c-11ac06c52c42/node-exporter/0.log" Jul 16 22:08:02.828832 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.828804 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-klcvk_a36431c2-8598-4a45-b16c-11ac06c52c42/kube-rbac-proxy/0.log" Jul 16 22:08:02.855035 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:02.855010 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-klcvk_a36431c2-8598-4a45-b16c-11ac06c52c42/init-textfile/0.log" Jul 16 22:08:03.004606 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.004573 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-c5b5n_cf548327-8c0f-4961-8c7b-d547ff8a9426/kube-rbac-proxy-main/0.log" Jul 16 22:08:03.034852 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.034823 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-c5b5n_cf548327-8c0f-4961-8c7b-d547ff8a9426/kube-rbac-proxy-self/0.log" Jul 16 22:08:03.063971 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.063881 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-c5b5n_cf548327-8c0f-4961-8c7b-d547ff8a9426/openshift-state-metrics/0.log" Jul 16 22:08:03.230747 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.230716 2577 generic.go:356] "Generic (PLEG): container finished" podID="e7e53153-a08e-43f9-abf9-e23473a54c04" containerID="2e31249c1277c4d3453b46d2ffc6e7164fb23ac4f4bdfe1e06147f5e281b1dad" exitCode=6 Jul 16 22:08:03.231318 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.230778 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" event={"ID":"e7e53153-a08e-43f9-abf9-e23473a54c04","Type":"ContainerDied","Data":"2e31249c1277c4d3453b46d2ffc6e7164fb23ac4f4bdfe1e06147f5e281b1dad"} Jul 16 22:08:03.231318 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.230837 2577 scope.go:117] "RemoveContainer" containerID="cec7f6d652581aff5b7ab079fb16c03e3d26eb175a2ad24fae06d502cf03af08" Jul 16 22:08:03.231318 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.231129 2577 scope.go:117] "RemoveContainer" containerID="2e31249c1277c4d3453b46d2ffc6e7164fb23ac4f4bdfe1e06147f5e281b1dad" Jul 16 22:08:03.231515 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:08:03.231422 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2_odh-ai-gateway-infra(e7e53153-a08e-43f9-abf9-e23473a54c04)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" podUID="e7e53153-a08e-43f9-abf9-e23473a54c04" Jul 16 22:08:03.268687 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.268648 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2"] Jul 16 22:08:03.348937 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.348862 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-flcfc_92605663-a637-4721-b2a9-c37ea6873763/prometheus-operator-admission-webhook/0.log" Jul 16 22:08:03.383761 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.383727 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7ccdc67968-mbhw7_9c6934bf-761d-4d75-86e3-8f74bdb8ffb9/telemeter-client/0.log" Jul 16 22:08:03.414750 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.414725 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7ccdc67968-mbhw7_9c6934bf-761d-4d75-86e3-8f74bdb8ffb9/reload/0.log" Jul 16 22:08:03.446256 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.446228 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7ccdc67968-mbhw7_9c6934bf-761d-4d75-86e3-8f74bdb8ffb9/kube-rbac-proxy/0.log" Jul 16 22:08:03.857562 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.857527 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx"] Jul 16 22:08:03.862443 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.862414 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:03.866387 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.866368 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bq49w\"/\"default-dockercfg-hm25x\"" Jul 16 22:08:03.882930 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.882897 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx"] Jul 16 22:08:03.899899 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.899866 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1e4c9f9-3c58-4db2-8af9-879fbcd81881-podres\") pod \"perf-node-gather-daemonset-ckxzx\" (UID: \"f1e4c9f9-3c58-4db2-8af9-879fbcd81881\") " pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:03.899899 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.899906 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1e4c9f9-3c58-4db2-8af9-879fbcd81881-sys\") pod \"perf-node-gather-daemonset-ckxzx\" (UID: \"f1e4c9f9-3c58-4db2-8af9-879fbcd81881\") " pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:03.900147 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.899943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1e4c9f9-3c58-4db2-8af9-879fbcd81881-lib-modules\") pod \"perf-node-gather-daemonset-ckxzx\" (UID: \"f1e4c9f9-3c58-4db2-8af9-879fbcd81881\") " pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:03.900147 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.899971 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wjl7\" (UniqueName: \"kubernetes.io/projected/f1e4c9f9-3c58-4db2-8af9-879fbcd81881-kube-api-access-2wjl7\") pod \"perf-node-gather-daemonset-ckxzx\" (UID: \"f1e4c9f9-3c58-4db2-8af9-879fbcd81881\") " pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:03.900147 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:03.900031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1e4c9f9-3c58-4db2-8af9-879fbcd81881-proc\") pod \"perf-node-gather-daemonset-ckxzx\" (UID: \"f1e4c9f9-3c58-4db2-8af9-879fbcd81881\") " pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:04.001416 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.001372 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjl7\" (UniqueName: \"kubernetes.io/projected/f1e4c9f9-3c58-4db2-8af9-879fbcd81881-kube-api-access-2wjl7\") pod \"perf-node-gather-daemonset-ckxzx\" (UID: \"f1e4c9f9-3c58-4db2-8af9-879fbcd81881\") " pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:04.001615 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.001484 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1e4c9f9-3c58-4db2-8af9-879fbcd81881-proc\") pod \"perf-node-gather-daemonset-ckxzx\" (UID: \"f1e4c9f9-3c58-4db2-8af9-879fbcd81881\") " pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:04.001615 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.001535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1e4c9f9-3c58-4db2-8af9-879fbcd81881-podres\") pod \"perf-node-gather-daemonset-ckxzx\" (UID: \"f1e4c9f9-3c58-4db2-8af9-879fbcd81881\") " pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:04.001615 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.001564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1e4c9f9-3c58-4db2-8af9-879fbcd81881-sys\") pod \"perf-node-gather-daemonset-ckxzx\" (UID: \"f1e4c9f9-3c58-4db2-8af9-879fbcd81881\") " pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:04.001792 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.001629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1e4c9f9-3c58-4db2-8af9-879fbcd81881-lib-modules\") pod \"perf-node-gather-daemonset-ckxzx\" (UID: \"f1e4c9f9-3c58-4db2-8af9-879fbcd81881\") " pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:04.001853 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.001807 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1e4c9f9-3c58-4db2-8af9-879fbcd81881-lib-modules\") pod \"perf-node-gather-daemonset-ckxzx\" (UID: \"f1e4c9f9-3c58-4db2-8af9-879fbcd81881\") " pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:04.002222 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.002172 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1e4c9f9-3c58-4db2-8af9-879fbcd81881-proc\") pod \"perf-node-gather-daemonset-ckxzx\" (UID: \"f1e4c9f9-3c58-4db2-8af9-879fbcd81881\") " pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:04.002363 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.002259 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1e4c9f9-3c58-4db2-8af9-879fbcd81881-podres\") pod \"perf-node-gather-daemonset-ckxzx\" (UID: \"f1e4c9f9-3c58-4db2-8af9-879fbcd81881\") " pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:04.002544 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.002514 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1e4c9f9-3c58-4db2-8af9-879fbcd81881-sys\") pod \"perf-node-gather-daemonset-ckxzx\" (UID: \"f1e4c9f9-3c58-4db2-8af9-879fbcd81881\") " pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:04.015022 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.014986 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wjl7\" (UniqueName: \"kubernetes.io/projected/f1e4c9f9-3c58-4db2-8af9-879fbcd81881-kube-api-access-2wjl7\") pod \"perf-node-gather-daemonset-ckxzx\" (UID: \"f1e4c9f9-3c58-4db2-8af9-879fbcd81881\") " pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:04.172371 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.172337 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:04.387716 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.384638 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx"] Jul 16 22:08:04.449956 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.449929 2577 scope.go:117] "RemoveContainer" containerID="02f0170a98621ba4bc3ecac71575f698718f95dcbbb45ecb29cca7cab7eaa770" Jul 16 22:08:04.468356 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.468327 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" Jul 16 22:08:04.507978 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.507936 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e53153-a08e-43f9-abf9-e23473a54c04-maas-api-service-ca\") pod \"e7e53153-a08e-43f9-abf9-e23473a54c04\" (UID: \"e7e53153-a08e-43f9-abf9-e23473a54c04\") " Jul 16 22:08:04.509387 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.509340 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e53153-a08e-43f9-abf9-e23473a54c04-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e7e53153-a08e-43f9-abf9-e23473a54c04" (UID: "e7e53153-a08e-43f9-abf9-e23473a54c04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:08:04.610173 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:04.610137 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e53153-a08e-43f9-abf9-e23473a54c04-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:08:05.249701 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:05.249672 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" Jul 16 22:08:05.249895 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:05.249705 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2" event={"ID":"e7e53153-a08e-43f9-abf9-e23473a54c04","Type":"ContainerDied","Data":"53396df61073d0236647fee66b4db6868a16a895201d37d1d1e76c520ba92a05"} Jul 16 22:08:05.249895 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:05.249776 2577 scope.go:117] "RemoveContainer" containerID="2e31249c1277c4d3453b46d2ffc6e7164fb23ac4f4bdfe1e06147f5e281b1dad" Jul 16 22:08:05.252797 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:05.252769 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" event={"ID":"de4e257c-afc4-4e28-a2fd-a1ab1de74638","Type":"ContainerStarted","Data":"fa439be37932f02d158c2edd3c6eb90063a3dce6436ee442a26ae9afe29f5ad5"} Jul 16 22:08:05.254778 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:05.254752 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" event={"ID":"f1e4c9f9-3c58-4db2-8af9-879fbcd81881","Type":"ContainerStarted","Data":"9e1a6aeb61c3344c61d185c9af3242b6111b68b2951fdd159179d6b83120e3e4"} Jul 16 22:08:05.254961 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:05.254800 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" event={"ID":"f1e4c9f9-3c58-4db2-8af9-879fbcd81881","Type":"ContainerStarted","Data":"e46b02e0a18b9e7f6529fda1ba53d6843bb2fd547e53ec38e509abcb8fc168e5"} Jul 16 22:08:05.254961 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:05.254834 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:05.312426 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:05.312372 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2"] Jul 16 22:08:05.323026 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:05.322966 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0f8cd113-s22b2"] Jul 16 22:08:05.346025 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:05.345963 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" podStartSLOduration=18.345942219 podStartE2EDuration="18.345942219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:08:05.342023494 +0000 UTC m=+3501.517131686" watchObservedRunningTime="2026-07-16 22:08:05.345942219 +0000 UTC m=+3501.521050414" Jul 16 22:08:05.397779 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:05.397721 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" podStartSLOduration=2.397705745 podStartE2EDuration="2.397705745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:08:05.394022768 +0000 UTC m=+3501.569130959" watchObservedRunningTime="2026-07-16 22:08:05.397705745 +0000 UTC m=+3501.572813936" Jul 16 22:08:06.268030 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:06.267948 2577 generic.go:356] "Generic (PLEG): container finished" podID="de4e257c-afc4-4e28-a2fd-a1ab1de74638" containerID="fa439be37932f02d158c2edd3c6eb90063a3dce6436ee442a26ae9afe29f5ad5" exitCode=6 Jul 16 22:08:06.268030 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:06.267985 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" event={"ID":"de4e257c-afc4-4e28-a2fd-a1ab1de74638","Type":"ContainerDied","Data":"fa439be37932f02d158c2edd3c6eb90063a3dce6436ee442a26ae9afe29f5ad5"} Jul 16 22:08:06.268308 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:06.268096 2577 scope.go:117] "RemoveContainer" containerID="02f0170a98621ba4bc3ecac71575f698718f95dcbbb45ecb29cca7cab7eaa770" Jul 16 22:08:06.268495 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:06.268466 2577 scope.go:117] "RemoveContainer" containerID="fa439be37932f02d158c2edd3c6eb90063a3dce6436ee442a26ae9afe29f5ad5" Jul 16 22:08:06.268696 ip-10-0-137-133 kubenswrapper[2577]: E0716 22:08:06.268675 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6_odh-ai-gateway-infra(de4e257c-afc4-4e28-a2fd-a1ab1de74638)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" podUID="de4e257c-afc4-4e28-a2fd-a1ab1de74638" Jul 16 22:08:06.355731 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:06.355690 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6"] Jul 16 22:08:06.453124 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:06.453087 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7e53153-a08e-43f9-abf9-e23473a54c04" path="/var/lib/kubelet/pods/e7e53153-a08e-43f9-abf9-e23473a54c04/volumes" Jul 16 22:08:07.464754 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:07.464726 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" Jul 16 22:08:07.542216 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:07.542118 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4e257c-afc4-4e28-a2fd-a1ab1de74638-maas-api-service-ca\") pod \"de4e257c-afc4-4e28-a2fd-a1ab1de74638\" (UID: \"de4e257c-afc4-4e28-a2fd-a1ab1de74638\") " Jul 16 22:08:07.542522 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:07.542491 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4e257c-afc4-4e28-a2fd-a1ab1de74638-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "de4e257c-afc4-4e28-a2fd-a1ab1de74638" (UID: "de4e257c-afc4-4e28-a2fd-a1ab1de74638"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:08:07.643857 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:07.643813 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4e257c-afc4-4e28-a2fd-a1ab1de74638-maas-api-service-ca\") on node \"ip-10-0-137-133.ec2.internal\" DevicePath \"\"" Jul 16 22:08:08.086132 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:08.086097 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zzmpt_3aa3fecc-3119-4df7-84c5-0ffcaf0f9606/dns/0.log" Jul 16 22:08:08.111392 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:08.111356 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zzmpt_3aa3fecc-3119-4df7-84c5-0ffcaf0f9606/kube-rbac-proxy/0.log" Jul 16 22:08:08.165129 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:08.165102 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bb7qw_eda93f33-be42-4a44-ab7c-1c1e93d65b74/dns-node-resolver/0.log" Jul 16 22:08:08.287700 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:08.287658 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" event={"ID":"de4e257c-afc4-4e28-a2fd-a1ab1de74638","Type":"ContainerDied","Data":"98bb158ac1a2bc3470a9b33541d7dae014e4dd2acda164ed02a27de5f52542f3"} Jul 16 22:08:08.287895 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:08.287753 2577 scope.go:117] "RemoveContainer" containerID="fa439be37932f02d158c2edd3c6eb90063a3dce6436ee442a26ae9afe29f5ad5" Jul 16 22:08:08.287895 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:08.287892 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6" Jul 16 22:08:08.317661 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:08.317628 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6"] Jul 16 22:08:08.323523 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:08.323498 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc2328dc-rgvh6"] Jul 16 22:08:08.453319 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:08.453283 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de4e257c-afc4-4e28-a2fd-a1ab1de74638" path="/var/lib/kubelet/pods/de4e257c-afc4-4e28-a2fd-a1ab1de74638/volumes" Jul 16 22:08:08.796402 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:08.796322 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7cd4df479d-m2fwl_1934850b-46ad-4994-9166-22aa589e1a0d/registry/0.log" Jul 16 22:08:08.847103 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:08.847073 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bcb4c_1d14ceb5-180d-4cd4-9653-70954a4f935b/node-ca/0.log" Jul 16 22:08:09.934929 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:09.934899 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4pfstf_eb922b04-96b8-401f-8610-017771c0b3b8/istio-proxy/0.log" Jul 16 22:08:10.181388 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:10.181357 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-585pl_42846e1e-6fd3-4bc3-9ec2-c558feb88415/discovery/0.log" Jul 16 22:08:11.154860 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:11.154831 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dx929_b4c047a7-f534-4e17-ac3b-acb669c58df3/serve-healthcheck-canary/0.log" Jul 16 22:08:11.280446 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:11.280415 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bq49w/perf-node-gather-daemonset-ckxzx" Jul 16 22:08:11.796820 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:11.796788 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pjlc6_f4e81032-aa8b-4011-a300-24472e2b6ab1/kube-rbac-proxy/0.log" Jul 16 22:08:11.817527 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:11.817503 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pjlc6_f4e81032-aa8b-4011-a300-24472e2b6ab1/exporter/0.log" Jul 16 22:08:11.842258 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:11.842230 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pjlc6_f4e81032-aa8b-4011-a300-24472e2b6ab1/extractor/0.log" Jul 16 22:08:14.777936 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:14.777907 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4pfstf_eb922b04-96b8-401f-8610-017771c0b3b8/istio-proxy/0.log" Jul 16 22:08:15.011675 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:15.011641 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-585pl_42846e1e-6fd3-4bc3-9ec2-c558feb88415/discovery/0.log" Jul 16 22:08:15.354678 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:15.354647 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9f7d7dc89-6jjxn_f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3/manager/0.log" Jul 16 22:08:15.413951 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:15.413866 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9f7d7dc89-6jjxn_f2b2bde4-3a9c-4891-9d9b-dbe5182e5eb3/copy-manifests/0.log" Jul 16 22:08:16.456621 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:16.456586 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4pfstf_eb922b04-96b8-401f-8610-017771c0b3b8/istio-proxy/0.log" Jul 16 22:08:16.635071 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:16.635039 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-585pl_42846e1e-6fd3-4bc3-9ec2-c558feb88415/discovery/0.log" Jul 16 22:08:16.767064 ip-10-0-137-133 kubenswrapper[2577]: I0716 22:08:16.766987 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68bcfc7f9d-kvkhl_8fb8205a-b1e8-424f-b3f0-a56664002770/manager/0.log"